Para-Gaussian Approximations of Random Deep Neural Networks

Abstract

It is known that randomly initialized neural networks have Gaussian behaviour in the $n\to \infty$ limit. Unfortunately, this approximation can be poor for finite $n$ and gets worse as one goes deeper into the network. In this work, we develop a more accurate approximation that is \emph{slightly} non-Gaussian, which we call the para-Gaussian distribution and is constructed from the Hermite polynomials. We show how this para-Gaussian approximation can lead to a practical recursive method to approximate the deeper layers of the network. Finally, we apply this theory to linear neural networks (where things can be solved exactly) and to ReLU neural networks (where things can be approximated).

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Machine Learning

• arXiv preprint · 27 October 2026

  Underdamped Anchored Langevin Dynamics

  Qi Feng, Hoang M. Nguyen and Lingjiong Zhu
• arXiv preprint · 28 October 2026

  Averaged Annealed Markov Chains Achieve the Algorithmic Threshold for Tensor PCA, Part II: Langevin Dynamics

  Aniket Das and Tselil Schramm
• arXiv preprint · 27 October 2026

  Risk measures for infinite-server queues driven by a Cox-BESQ process

  José Da Fonseca, Yajun Shi and Wenjun Zhang
• arXiv preprint · 28 October 2026

  Capacity of the Hopfield-Hebbian dynamics with correlated patterns

  Mihailo Stojnic
• arXiv preprint · 27 October 2026

  Holomorphic Cauchy Networks for Efficient Stationary Tail Computation in High-Dimensional Reflected Brownian Motions

  Kehan Wang

All new papers in Machine Learning