Do Neural Models Prefer Their Submodels?

Abstract

A neural model is the set of functions that can be expressed by a given neural network architecture. Neural models contain natural submodels, such as those obtained by setting some neurons to be inactive. We study the geometry of neural models and ask whether they prefer their submodels. By this we mean whether local minimizers of the loss function lie in a given submodel. We introduce a geometric framework to determine if this occurs. For shallow multilayer perceptrons, we show that subnetworks with inactive hidden neurons are never preferred. For deep multilayer perceptrons, we give conditions under which subnetworks with inactive hidden neurons are not preferred. For lightning self-attention, lower-rank submodels are usually not preferred, but we identify examples where they are because they lie on a boundary of the model. Our results show that, aside from a few exceptions, preference for submodels cannot be explained by the geometry of the neural model, and must instead come from other sources such as training dynamics.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Machine Learning

• arXiv preprint · 21 October 2026

  On the gradient conjecture at infinity: Finite spherical length for gradient trajectories of definable potentials

  Thang Do, Arnulf Jentzen and Benno Kuckuck
• arXiv preprint · 20 October 2026

  Generic finiteness of solutions to parametrized systems of generalized polynomial inequalities

  Stefan Müller
• arXiv preprint · 21 October 2026

  Geometric knowledge spaces, Attention pairing and Architectures

  Shang Xu
• arXiv preprint · 18 October 2026

  Semi-supervised learning on singular data sets

  Guillaume Valette
• arXiv preprint · 23 October 2026

  Structure and Exact Certification of Nonnegative Tensor Decompositions

  Haoming Wang and Ming Yuan

All new papers in Machine Learning