arXiv preprint ·
A Spectral Renormalization-Group View of Learning
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
We study how the layers in deep neural networks (NNs) converge, using Wilsonian Renormalization-Group (RG) ideas to organize the spectral picture. During training, these layers often develop heavy-tailed power-law (PL) spectra, and the strongest checkpoints often appear near the PL exponent $α\approx 2$. This pattern raises a simple question: why should this value mark a well-trained layer? The answer is both statistical and physical. Our statistical hypothesis is that $α=2$ marks the boundary between stable if not optimal learning and overfitting via non-self-averaging. The physical answer is that the same value makes the spectral energy gain scale evenly across logarithmic spectral scales, which is the proposed Wilsonian marginal condition. We then apply the ERG trace-log condition, the proposed SETOL theory, which we justify by taking volume-preserving, scale-invariant transformations to remove arbitrary layer scale along the proposed RG flow. The resulting spectral RG flow connects the HTSR and SETOL theories to optimizers such as Muon, and motivates a new optimizer extension, WW-PGD. Together, these results point toward a spectral Renormalization-Group view of learning, in which NNs may approach shared spectral structure in the settings studied here.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Machine Learning
Distilling Knowledge from Search: Decoder Model Factory for Quantum LDPC Codes
Solvable dynamics model of in-context learning
Characterization of Phase Transitions with the Annealed Importance Sampling Algorithm
How Does Chain-of-Thought Supervision Teach Models to Reason? A Theoretical Analysis
Generative diffusion in long-range correlated landscapes: Speciation cascade and the foam structure of the score
Get the few papers worth your time in Machine Learning every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.