Continual Classification Beyond the Kernel Regime: Harnessing the Sparsity Bias of L2 Regularization in Diagonal Linear Networks

Abstract

We study logistic regression for continual learning with linear predictors in a high-dimensional statistical setting. Existing theoretical work has largely focused on continual linear regression with variants of L2 regularization, motivated by their analytical simplicity and connection to the kernel regime of neural networks. We first derive the optimal L2 regularization strength for continual linear classification and characterize its generalization performance, recovering behavior analogous to prior results for continual linear regression. Next, we examine the effect of L2 regularization in deeper models using a minimal two-layer diagonal linear network. In the single-task setting, L2 regularization on the weights of such models is known to promote sparsity in the resulting predictor, yielding a rich, sparse feature-learning regime beyond the kernel regime (a similar effect arises in two-layer ReLU networks). However, in continual learning, where L2 regularization is typically centered on previous-task weights, this sparsity bias is no longer preserved. We therefore propose a weighted variant of L2 regularization that extends this beneficial sparsity bias to continual learning. We show theoretically and empirically that, for a sufficiently sparse teacher, our variant converges substantially faster than directly learning a linear predictor with conventional continual L2 or L1 regularization, trading some long-horizon bias for markedly improved performance over moderate task horizons.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Machine Learning

• arXiv preprint · 4 November 2026

  Optimizing Recourse Under Predictive Multiplicity

  Shahin Jabbari
• arXiv preprint · 5 November 2026

  Using Transformers for Teaching and Learning via Symbolic Distillation of Neural A-life Agents

  Shuwei Wang, Vadim Bulitko, Matthew R. G. Brown and 5 others
• arXiv preprint · 4 November 2026

  Timing is Everything: Inferring Temporal Membership in Continually Updated Models

  Jie Fu, Xiangpeng Deng, Yuan Hong and 1 other
• arXiv preprint · 5 November 2026

  TriAttention: Unified Sparse Mixed-Precision Attention for Long-Context LLM Inference

  Hanzhi Zhang, Qiao Zhang, Qinglei Cao and 4 others
• arXiv preprint · 4 November 2026

  Unsupervised Neural Operators for Rotating Relativistic Stars: High-Fidelity Equation of State Inference and Pulsar Spin Evolution at Scale

  Anarya Ray

All new papers in Machine Learning