arXiv preprint ·
Deep ReLU Networks Can Fail the Best Approximation Property for Continuous Target Functions
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
We construct a continuous function on $[-1,1]^2$ for which the infimum of the $L^2$-approximation error over the class of ReLU networks with two hidden layers and architecture $(2,2,1,1)$ is not attained. This counterexample shows that deep ReLU networks do not satisfy the best approximation property in general, even for continuous target functions. Our findings contrast with existing results for shallow neural networks, where ReLU is one of the few activation functions that guarantee the existence of global minima in typical regression tasks. The proof is based on the fact that the breaklines for neurons in the second hidden layer can bend, which breaks the symmetry arguments used in the one-hidden-layer case.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Machine Learning
Covariance-Informed Ensemble Score Filtering: Embedding Localized Covariance Geometry into Score-Based Transport
Convex Equilibrium Potentials for Monge--Ampère: Curvature Bounds and A Posteriori Evaluation
Solving and Learning Navier-Stokes Dynamics with the Neural Basis Method
Structure-preserving reduced-order models of thin panels in supersonic and hypersonic flow
CoPiT: A conditional Position-induced Transformer for aerodynamic field prediction on unstructured meshes
Get the few papers worth your time in Machine Learning every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.