Width and Depth Limits Commute During Training in Linear Residual Networks

Abstract

Recent work established that the infinite-width and infinite-depth limits of residual networks commute at initialization when residual branches are scaled by $1/\sqrt{depth}$. In particular, the network output converges to the same limit whether width and depth tend to infinity sequentially or jointly. We extend this commutativity result to training in linear residual networks. Under the same $1/\sqrt{depth}$ residual-branch scaling (Depth-$μ$P) and the $μ$P width parameterization, we prove that, for any fixed number of gradient-descent steps with gradient clipping, the network output converges in $L^2$ along every joint width--depth growth path as both width and depth tend to infinity. Consequently, the infinite-width and infinite-depth limits commute throughout any finite training horizon. Without gradient clipping, the corresponding $L^2$ convergence does not hold in general, but we establish the weaker convergence in probability.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Machine Learning

• arXiv preprint · 22 October 2026

  Lifetime-valid decisions by continually evolving AI analysts

  Xiaojing Sun, Yuhan Zeng and Xiao Wang
• arXiv preprint · 23 October 2026

  Rethinking Representation Geometry in Supervised Contrastive Learning under Long-Tailed Data

  Joonhyeok Jeong, Jy-yong Sohn and Kibok Lee
• arXiv preprint · 22 October 2026

  Residual Calibration via Local Feature-Space Refinement

  Xihao Wang, Chengyao Yu and Ruixing Ming
• arXiv preprint · 23 October 2026

  DBCM: Dual-Branch Calibration Module for Test-Time Adaptation in Time Series Forecasting

  Jaebyeong Jeon, Kunwoo Kang, Sang Yup Lee and 1 other
• arXiv preprint · 23 October 2026

  Conditional Flow Models for Offline Maximum Entropy Reinforcement Learning

  Hao Cheng, Yuling Jiao, Lican Kang and 2 others

All new papers in Machine Learning