arXiv preprint ·
Affine Geometry of Gaussian ReLU Networks via Conditional Kac-Rice Formulas
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
We study how the affine geometry of finite ReLU networks is created at random initialization and reorganized by supervised training. We call a sign-changing zero of a hidden preactivation an activation switch and a point where the scalar network output is nondifferentiable a scalar kink. For one-dimensional input, conditioning on the preceding layers makes each preactivation Gaussian and affine on the cells of a random finite partition. This yields an exact finite-width conditional Kac-Rice formula for the expected number of activation switches along an input interval. For fixed depth and proportionally growing widths, the resulting switch intensities converge to explicit deterministic limits. A visibility estimate shows that the expected number of switches that do not produce scalar kinks is negligible, yielding an explicit leading formula for the expected number of scalar kinks and hence affine regions. In higher input dimensions d >= 2, the analogous conditional surface formula yields the leading expected (d-1)-dimensional Hausdorff measure of the scalar kink set. On the Breast Cancer Wisconsin data, the initialization formula accurately predicts switch counts along held-out segments. After training, switch counts decrease along within-class segments and increase along between-class segments. Thus, training redistributes rather than merely contracts affine complexity.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Machine Learning (Statistics)
Non-Adaptive Learning of Sparse Erdős--Rényi Graphs via Affine Splitting
Domain Adaptation with Target Information via Doubly-Anchored Distributionally Robust Optimization
Probabilistic Geodesic Flow Matching on Location-Scale Families
On the asymptotic shape of quantile surfaces
Structured Neural SDEs for Functional Calibration
Get the few papers worth your time in Machine Learning (Statistics) every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.