arXiv preprint ·
Hierarchical Prototype Emergence in Modern Hopfield Models
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Hierarchical correlations are a universal feature of any realistic model of data, and the question of how associative memory models may learn these correlations and generalize beyond them to construct new sensible images is an important step towards understanding more complex modern architectures such as diffusion models. We consider a hierarchical model for memories which are sampled and stored in a dense Hopfield network with polynomial activation. We analytically derive conditions for each level of this hierarchy to be locally stable - that is they are local energy minima. We use prototype reconstruction as a minimal model of generalization and we find that it takes only a quasi-polynomial amount of information to generalize beyond particular memories and even particular groups in the hierarchy. We observe a qualitatively analogous phase diagram in the number of memories, sharpness of the activation function (polynomial degree) for data from Fashion-MNIST.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Disordered Systems & Neural Networks
Phases in a class of associative memories via hidden neurons
Bridging Control, Inference, Transport, and Thermodynamics: From Theory to Applications in Learning
Learning structural balance of graphs from quantum spectral features
A Data-free Universal Prior over Syntactic Structures
Bias-Induced Crossover in Absolute Capacity of Dense Associative Memory
Get the few papers worth your time in Disordered Systems & Neural Networks every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.