arXiv preprint ·
Softmax-Free Attention in Spiking Transformers as the Zero-Temperature Limit of Two-Level Systems
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Spiking Transformers achieve energy efficiency by discarding softmax normalization in favor of thresholded, binary attention. However, this removal breaks the thermodynamic framework used to analyze attention networks, as the normalized measure and partition function cease to exist. We restore this theoretical foundation by proving that softmax-free spiking attention operates exactly as the zero-temperature limit of a product of independent two-level systems. This correspondence places spiking attention strictly in the frozen ordered phase of the thermodynamic phase diagram for all inputs. Consequently, the entropic cost of organizing attention collapses, replaced by a Landauer-type physical floor governed by action-potential generation. Furthermore, this ordered-phase pinning mathematically explains the optimization difficulty caused by vanishing surrogate gradients. We validate this theory with forward-relaxation experiments, placing both standard and neuromorphic attention mechanisms on a unified thermodynamic axis.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Machine Learning
Explicit Bounds on Hoffman Constants for Sparse Polyhedral Regularizers
A 4-Qubit Variational Quantum Classifier for Sequence-Specific Classification of DNMT3A and DNMT3B Flanking Methylation Preferences
EvolvAttack: Backdooring self Evolving VLM Agents
Improving the Evaluation of Biomedical Question Answering with Few Human Preference Signals
ObsSpec: Accelerating Agentic Reinforcement Learning with Observation Speculation
Get the few papers worth your time in Machine Learning every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.