Sharp Generalization for Nonparametric Regression in Interpolation Space by Shallow Over-Parameterized Neural Networks with Channel Attention

Abstract

We study nonparametric regression using an over-parameterized two-layer ReLU neural network with fixed covariance-based channel attention, trained by gradient descent with both layers updated simultaneously. The spherical covariates follow an arbitrary fixed non-atomic distribution, and the target function lies in an interpolation space with spectral bias widely studied in the statistical learning literature. We prove that early-stopped training achieves regression risk $\cO(\eps_n^2)$, where $\eps_n$ is the critical population rate of the attention kernel $\Kattn$. Its positive eigenvalues are $λ_j^{s'}$, where $λ_j$ are those of the ReLU feature covariance kernel $K$ and $s'=2s+3$ for $s\ge1$. Under polynomial eigenvalue decay $λ_j\asymp j^{-2α}$ with $α>1/2$, the regression risk bound $\cO(\eps_n^2)$ specializes to the minimax optimal rate $\cO(n^{-2αs'/(2αs'+1)})$. Our analysis is based on two new technical results on the optimization and generalization of the attention network. First, the neural network function after every gradient descent iteration admits a decomposition into a function with bounded Reproducing Kernel Hilbert Space (RKHS) norm and a uniformly small residual. Second, local Rademacher complexity converts this decomposition into a regression risk bound. Kernel gradient descent on the same ball attains the same rate, while our theorem gives explicit sufficient width and auxiliary-sample conditions for simultaneous network training. Our analysis is further extended to covariates supported on $\RR^d$ through a novel normalization method. Under a sub-Gaussian covariate distribution, the attention network achieves a nearly optimal regression risk $\cO((\log n)n^{-s'(d+3)/(s'(d+3)+d)})$ without a density assumption.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Machine Learning (Statistics)

• arXiv preprint · 1 November 2026

  Variational learning of energy-stable macroscopic flow equations from microscopic laws

  Pei Ge and Huan Lei
• arXiv preprint · 2 November 2026

  On Stochastic Continuous-Time Linear System Identification for Adaptive Stabilization

  Reza Sadeghi Hafshejani and Mohamad Kazem Shirani Faradonbeh
• arXiv preprint · 1 November 2026

  Transport for Conditional Sampling: Foundations and Applications

  Ricardo Baptista, Benjamin Bloem-Reddy and Bamdad Hosseini
• arXiv preprint · 2 November 2026

  Efficient Proper Agnostic Learning of Halfspaces

  Anindya De, Elchanan Mossel, Shivam Nadimpalli and 1 other
• arXiv preprint · 1 November 2026

  Correctness and Predictive Uncertainty in Bayesian TSK Fuzzy Regression: A Reproducible Evaluation on Tabular and Building-Meter Data

  Zhiren Xiao

All new papers in Machine Learning (Statistics)