Mesoscopic learning dynamics of extensive-width neural networks

Abstract

Understanding how stochastic gradient descent learns internal representations in wide neural networks trained on rich target functions is a central problem in machine learning. We study this question in two-layer neural networks learning target functions represented by networks of the same architecture, with arbitrary nonlinearity, and with network width and input dimension growing proportionally. In this regime, standard approaches for describing the learning dynamics require tracking a diverging number of overlaps, which measure correlations between learned and target features, leaving no obvious description in terms of a few order parameters. Here, we develop a mesoscopic description of learning by characterizing the distribution of overlaps. In the early stages of learning, we find that a Gaussian description of overlap statistics reduces the learning dynamics to a few collective variables that track learning of the target function's linear and quadratic components. We then estimate the timescale of specialization, when Gaussian closure breaks down and individual learned features begin to strongly correlate with specific target features. We find that specialization in extensive-rank networks is accelerated via a quadratic shielding mechanism, whereby learning the linear and quadratic components of the target function filters gradient fluctuations. We finally describe how the activation function determines the pattern of specialization and characterize the subsequent refinement of learned features. Together, these results show how collective learning gives way to individual feature discovery and clarify when learning in large neural networks admits a simple statistical description.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Machine Learning (Statistics)

• arXiv preprint · 16 October 2026

  Towards optimal selection schedules for online active learning

  Hugo Cui, Francesca Mignacco and Francesco Mori
• arXiv preprint · 17 October 2026

  Power-Law Correlations Without Apparent Temperature-Tuned Criticality in Fixed-Tree Hierarchical Languages

  Marcel Kühn and Bernd Rosenow
• arXiv preprint · 18 October 2026

  Higher-order disorder correlations control reflected phonon counting statistics

  Ara Sedrakyan
• arXiv preprint · 16 October 2026

  Sampling the XY model with an attention-based neural network

  Piotr Białas, Christian D. Morales, Tomasz Stebel and 1 other
• arXiv preprint · 19 October 2026

  Dynamical Batching: How Class-Cycling Training Shapes the Internal Structure of Neural Networks

  Nicolás Ratier-Werbin, Samuel Lozano and Miguel Ruiz-García

All new papers in Machine Learning (Statistics)