Statistical and Privacy Guarantees for Deep M-Regression via Stochastic Gradient Langevin Dynamics

Abstract

In this article, we study deep nonparametric regression trained via stochastic gradient Langevin dynamics (SGLD), a variant of stochastic gradient descent that injects carefully calibrated Gaussian noise during optimization. Using a PAC-Bayesian analysis tailored to deep ReLU networks, we establish an excess-risk bound that simultaneously accounts for network approximation error, statistical complexity, and the algorithmic cost accumulated along the SGLD trajectory. Under a Bernstein condition and within a stable SGLD regime, the excess-risk bound contains a stochastic term whose coefficient is inversely proportional to the variance of injected noise, while the estimator retains the optimal rate for nonparametric regression under the Hölder smoothness condition up to logarithmic factors. This result provides an approximation-error-aware, rate-optimal statistical perspective on the widely observed generalization benefits of moderate SGLD noise. For per-sample clipped SGLD, we further establish an exact f-DP guarantee and the corresponding asymptotic $μ$-GDP guarantee for the proposed estimator. We establish statistical convergence rates and differential privacy guarantees for DNN estimators trained using SGLD, while explicitly accounting for both network approximation error and the effect of injected noise in the excess-risk bound. Extensive simulation studies and analyses of two real datasets examine the finite-sample predictive performance and privacy-utility behavior of the proposed procedure.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Methodology

• arXiv preprint · 5 November 2026

  Changepoint localization via nested exchangeability tests: from point estimates to confidence intervals

  Rohan Hore and Aaditya Ramdas
• arXiv preprint · 5 November 2026

  Adaptive Experimentation Can Hide Model Error:Detection Limits and Falsification Budget

  Ramnarayan Choudhary
• arXiv preprint · 5 November 2026

  Identification and Bounding of Joint Expectation over Potential Outcomes

  Yuta Kawakami and Jin Tian
• arXiv preprint · 5 November 2026

  Joint Modeling of Conditional Mean and Variability in Multilevel Functional Data: Application to Postprandial Glucose Trajectories in the AEGIS Study

  Yang Xiao, Francisco Gude and Marcos Matabuena
• arXiv preprint · 5 November 2026

  Doubly-robust causal inference for multi-valued treatment regimes with recurrent neural networks

  Jason Poulos

All new papers in Methodology