arXiv preprint ·
How Does Chain-of-Thought Supervision Teach Models to Reason? A Theoretical Analysis
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Training with chain-of-thought (CoT) supervision is an effective approach for teaching models multi-step reasoning procedures. At inference time, however, models may need to continue reasoning beyond the finite teacher CoT sequences observed during training, and longer reasoning can instead degrade performance, a phenomenon known as overthinking. How the amount and structure of training-time CoT give rise to such failures remains theoretically poorly understood. To study this problem, we introduce an in-context weight prediction model trained on finite gradient descent trajectories for linear regression. Using high-dimensional asymptotic analysis, we show that the success and failure modes of long-horizon reasoning undergo phase transitions depending on the context length and the total amount of training CoT. We further distinguish two mechanisms of overthinking, error accumulation and error amplification, and show that with short training contexts, increasing the training CoT budget can worsen long-horizon error before eventually stabilizing reasoning. Experiments with fully trained linear and multi-head softmax attention further show that these qualitative behaviors persist. Our results provide a theoretical account linking properties of training-time CoT to inference-time failures and offer guidance for designing teacher CoT that supports stable reasoning.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Machine Learning
Generative diffusion in long-range correlated landscapes: Speciation cascade and the foam structure of the score
Solvable dynamics model of in-context learning
Bulk and edge states in Fibonacci-Bose-Hubbard clusters
Distilling Knowledge from Search: Decoder Model Factory for Quantum LDPC Codes
Dynamical Signatures of Selective Gap Closing in Multi-Dimensional Monotone Quasiperiodic Operators
Get the few papers worth your time in Machine Learning every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.