arXiv preprint ·
The Role of Feed-Forward Layers in Transformer Dynamics
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
We study the dynamical behavior of tokens in transformers from a control-theoretic perspective. Our model includes the feed-forward layer present after the self-attention mechanism, with the self-attention mechanism interpreted as an interacting particle system and the feed-forward layer as an independent control. Our main theoretical result establishes that the feed-forward network can steer the tokens arbitrarily close to consensus regardless of the key, query, and value matrices. Our result are easily extended to convergence to many clusters and to multi-head attention. We conduct numerical experiments to verify our results, and compare thresholding behavior from our theory to real-world LLMs.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Machine Learning
DisKO: Deep Koopman Learning in Distribution Space from Unpaired Snapshots
Identifiability Guarantees for Drivers and Dynamics of Delayed Physical Systems
Dynamics to decision: A mathematical theory of Lyapunov spectra and decision boundaries in deep classifiers
Prediction Limits and Koopman Closure of Geometry-Induced Soft State Abstractions
TopTimeNet: Topologically-assisted time-series classification model
Get the few papers worth your time in Machine Learning every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.