Nonequilibrium Phases of Repulsive Self-Attention: Chaos, Attention Condensation, and Emergent Locality

Abstract

We study the nonequilibrium dynamics of a minimal recurrent transformer with $N$ normalized tokens, $Q=K=I$, and a negative value map $V=-I$. Similarity-based attention selects nearby representations, while the negative value map drives tokens away from the selected field. This feedback can continually reorganize both the representation geometry and the attention network. For $d=2$, the tokens lie on a circle, where the regular polygon is an exact fixed point. As the attention feedback strength $γ$ is increased, the polygon loses stability through a flip bifurcation, giving rise to period-two motion, chaos, and cluster-exchange or cluster-flip states. Despite this temporal complexity, attention remains diffuse as $N\to\infty$ at finite fixed softmax sharpness $β$. Attention condensation instead emerges in the scaling regime $β\sim N^2$. In the hard-routing limit, repulsive updates amplify local perturbations and routing-partner switches transmit them ballistically, producing an emergent butterfly cone in representation space. High-dimensional geometry provides a distinct route to localization. For $d=N\to\infty$, simulations from Gaussian initial conditions provide evidence for a condensation transition at $β=O(1)$, driven by dynamically generated finite overlap gaps. Depending on $γ$, the resulting phases include diffuse simplex-like states, consensus flips, condensed active routing with signatures of chaos, and fragmented cluster flips. These results establish temporal activity, attention condensation, and geometric clustering as distinct collective phenomena, and show that sparse attention can sustain persistent dynamics rather than freeze it.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Disordered Systems & Neural Networks

• arXiv preprint · 23 September 2026

  Discrete Diffusion Models via Evolving Variational Autoregressive Networks

  Kewen Pan and Ying Tang
• arXiv preprint · 24 September 2026

  Accurate Sampling from Diffusion Models

  Dénes Sexty
• arXiv preprint · 23 September 2026

  Optimal low-rank compression of quantum dynamics

  Hugo Mackay, Donghoon Kim, Tan Van Vu and 1 other
• arXiv preprint · 24 September 2026

  Disorder-induced quantum Fisher information in topological quantum systems

  Advay Burte, Keshav Das Agarwal, Leela Ganesh Chandra Lakkaraju and 1 other
• arXiv preprint · 23 September 2026

  Rank-One Signal Recovery in Sparse Wishart Noise

  Preben Forer, Urte Adomaityte and Pierpaolo Vivo

All new papers in Disordered Systems & Neural Networks