arXiv preprint ·
Riding the Fixed-Point Curve: Image Classification in the Continuous Thought Machine
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
The Continuous Thought Machine (CTM) is a recurrent network that attends to inputs and makes predictions based on the synchronization between its neurons. On tasks spanning many domains, it displays emergent thought processes that have not been explained. We report a number of findings about how the CTM classifies images, focusing on ImageNet. Thinking steps are attracting but strongly non-normal maps exhibiting balanced amplification (Murphy and Miller, 2009); while attraction is already present at CTM initialization, training highly promotes non-normality. By Banach's theorem, the neural state thus has a locally unique attracting fixed point, which moves at every step, and the thought trajectory is the curve these fixed points trace, followed one tick behind. Most of that curve's motion is shared across images, and visible as a previously reported traveling wave of activity across neurons. The attention schedule along the thought is largely generic: shuffling queries across images at every tick costs only 7 percent of classification accuracy, while image-independent increments predict the next query with R^2 = 0.98 (held out). Our results suggest that the CTM has found a single strategy to process all ImageNet images.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Machine Learning
Collective Benefits and Efficiency in Coupled Echo State Networks
How Long Is the Coast of a Neuron? Resolution-Dependent Interaction Order and Computational Capacity
Neural Dimensional Analysis: Foundations of Dimensionless and Scale-Consistent Computation
Design and Analysis of Temporal Representations in Neural Networks
Saliency-Guided Replay for Continual Learning in Deep Spiking Neural Networks
Get the few papers worth your time in Machine Learning every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.