arXiv preprint ·
Do Artificial Neural Networks Need Dendritic Nanotubes? Two Reasons for Shrinking Importance Estimates in Continual Learning
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Regularization-based continual learning protects each parameter in proportion to an estimate of its importance, yet these estimates come from finite samples and heuristics and are used as if they were exact. We report four studies that start from a biologically inspired mechanism and end with a statistical explanation of it. Motivated by dendritic nanotubes, we let transient lateral links diffuse importance tags between hidden units. Links between co-active units gave no significant gain, but random links with lifetimes of 25 or 200 steps raised final accuracy on 20-task Permuted MNIST by 1.5 to 2.6 points. We show that random diffusion implements linear shrinkage of importance toward the layer mean in expectation. A simulation of the link process alone predicts shrinkage weights of 0.23 and 0.52 for these lifetimes, and graph-free shrinkage at 0.25 and 0.5 reproduces both the accuracy and the dispersion of the consolidated importance, while every activity-dependent rule did worse than random links. A five-seed screen across EWC, SI, and MAS found nominal gains for SI and MAS on Permuted MNIST and losses on task-incremental Split CIFAR-100, where forgetting was small and shrinkage cost plasticity. Varying the number of importance samples then separates two mechanisms on Permuted MNIST. For EWC, the gain fell by about ten points per tenfold increase in samples, from 19.6 points with 10 samples to -0.4 with 1000, a variance effect. For MAS, a gain of about one point persisted at 1000 samples, consistent with a bias in the importance measure rather than noise. Shrinkage had no detectable effect for MAS on a domain-incremental CIFAR-100 benchmark. Finally, we show that a standard heteroscedastic empirical Bayes rule shrinks the largest Fisher-type estimates most, which may explain its poor results, and point to a log-scale alternative.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Machine Learning
The value of information and the emergence of communication in evolved e-puck robots
PotentialEP: Backprop-Free Training of CIFAR-Scale Generative Models via Equilibrium Propagation
REMBGO for Evaluation-Limited Optimization: CEC2022 Benchmarking and EAF-STATCOM Generalization
RELAGRAM: Recursive Latent Graph Model for the Capacitated Vehicle Routing Problem
A Multi-Agent LLM System for Automated Tuning of Specialized Neural Network Models
Get the few papers worth your time in Machine Learning every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.