arXiv preprint ·
Geometric knowledge spaces, Attention pairing and Architectures
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
In this paper, we propose a geometric approach to deriving attention mechanisms and exploring potential new transformer architectures. We begin by formalizing the notions of knowledge worlds and attention pairings, then demonstrate how this framework naturally leads to the architectures of standard attention, causal attention, linear attention, tensor attention, and recurrent neural networks. Finally, we introduce several new geometric spaces that may inspire novel neural network architectures.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Machine Learning
The Geometry of Limits in ReLU Function Spaces
Graph Representation via Elements of Discrete Morse and Cobordism Theories
On the gradient conjecture at infinity: Finite spherical length for gradient trajectories of definable potentials
Structure and Exact Certification of Nonnegative Tensor Decompositions
Polynomial Residual Networks are Generically Identifiable
Get the few papers worth your time in Machine Learning every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.