SympFormer: Inertial Attention Blocks from Damped Hamiltonian Dynamics on Spaces of Probability Measures

Abstract

The attention mechanism underpins modern large language models, yet their dynamics remain only partially understood. Recent perspectives interpret attention blocks as interacting particle systems, whose mean-field limits correspond to gradient flows of energy functionals on spaces of probability densities on $\mathbb R^d$ equipped with Wasserstein-type metrics. We extend this viewpoint by deriving accelerated attention blocks from inertial, Nesterov-type damped Hamiltonian dynamics on these density spaces, developing the geometry and differential calculus needed to make the construction rigorous. In particular, we introduce a local-mobility geometry for the Otto-Wasserstein and softmax-Wasserstein settings, and a quotient Stein geometry that resolves the degeneracy of the linear-attention kernel to yield a well-defined metric and differential calculus. We provide rigorous foundations when softmax attention admits a gradient-flow formulation and when its inertial counterpart admits a damped Hamiltonian on explicitly characterized domains. For linear attention, we obtain a finite-dimensional solution theory with sharp conditions for global existence versus finite-time second-moment blow-up. We further prove a quantitative link between continuous moment blow-up and the discrete-layer depth at which token norms cross a prescribed threshold. Finally, motivated by the finite-token geometry, we propose geometric corrections to token and momentum updates. In decoder-only, nanoGPT-style experiments on TinyStories and OpenWebText, these corrections lower validation negative log-likelihood on next-token prediction relative to momentum-only controls.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Symplectic Geometry

• arXiv preprint · 19 October 2026

  Non-planar contact structures with Calabi-Yau Fillings and topological quantum computation

  Atsuhide Mori
• arXiv preprint · 18 October 2026

  Integrable building blocks for symplectic neural networks: learned-type layers, fiber-preserving layers, and what a single layer can do

  Andrew Qing He
• arXiv preprint · 24 September 2026

  Planar Contact Structures with Calabi-Yau Fillings and Topological Quantum Computation

  Atsuhide Mori
• arXiv preprint · 17 September 2026

  Special Lagrangian cones in Deep Learning

  Tejas Kotwal and Govind Menon
• arXiv preprint · 26 October 2026

  Geometric aspects of fault tolerance

  Jonathan Conrad

All new papers in Symplectic Geometry