arXiv preprint ·
Speed Limit for Information Acquisition in Stochastic Learning Dynamics
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Neural networks acquire internal representations through learning. In this work, we formulate stochastic gradient descent (SGD) as a Markovian stochastic process and derive a Fisher-information flow speed limit that bounds the rate at which trainable parameters can acquire information about latent variables in the data-generating process. The resulting inequality decomposes the information flow into drift and noise contributions, thereby quantifying the roles of deterministic learning forces and SGD-induced fluctuations from an information-theoretic perspective. We verify the bound in analytically tractable basis-function linear regression, where the information budget predicted by the bound reproduces the ordering and characteristic time scales with which different latent variables are encoded in the learned parameters. These results establish Fisher-information speed limits as a quantitative framework for diagnosing when and how different aspects of the data-generating mechanism are acquired during stochastic learning.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Statistical Mechanics
Non-equilibrium dissipative stabilization of s- and d-wave superconductivity
Copying explains the collective behavior of AI agents in the wild
Might the radiation era extend back to the Big Bang? On dark matter production and the relic graviton background in quadratic gravity
Human mutation field reveals an equilibrium-like structure with irreversible circulation
Event-Time Order-Flow Memory, Operational-Time Impact, and Subordinated Market Observables
Get the few papers worth your time in Statistical Mechanics every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.