arXiv preprint ·
Dynamical coarse-graining of neural-network training
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Dynamical coarse-graining of neural networks gives macroscopic observables and enables training to be predicted, and controlled in a state space far smaller than the parameter space. In this paper, we first define the learning operator following the training law of neural networks and establish a descent theorem for the learning operator, which serves to determine whether the operator is sufficient to characterize the learning dynamics. Then the exact closure theorem and the dynamic coarse-graining theorem are presented in sequence. The three theorems established in this paper provide a systematic framework for constructing and assessing dynamical coarse-grained models of neural-network training. Finally, we demonstrate the applicability of the theorems with a finite-width nonlinear ReLU network. Its dynamic coarse-grained model for a fixed pattern is exactly derived and composed of five independent moments. The model exactly predict the training outputs and evolving learning operator. The framework may also guide the development of approximate coarse-grained models for more general neural networks when exact closure is unavailable.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Machine Learning
Distilling Large-Scale Machine Learning Interatomic Potentials to Predict Long-Timescale Electrolyte Transport Properties
Ideal interaction dynamics of defects and background fields in condensed matter
Hidden Phase Transition in Single-Polymer Stretching under Elongational Flow
Supercoiling Drives DNA-Protein Co-Condensation
Multistable nematic liquid crystal pores as reconfigurable optical vortex metasurfaces
Get the few papers worth your time in Machine Learning every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.