Decoupled and Distilled: Task-Adaptive LoRA-Teachers with Ensemble Knowledge Transfer for Few-Shot Class-Incremental Learning

Abstract

Few-Shot Class-Incremental Learning (FSCIL) addresses the challenge of learning new classes from very limited samples while retaining knowledge of previously learned ones. Although parameter-efficient fine-tuning methods with pre-trained models show promise for class-incremental learning, strict gradient-based constraints can be unreliable under severe data scarcity, while multi-expert approaches can impose substantial inference-time costs. We propose TALON (Task-Adaptive LoRA-Teachers with Ensemble Knowledge Transfer), an inference-efficient FSCIL framework. TALON dynamically allocates an independent LoRA-Teacher to each incremental task for task-specific representation learning, then distills multiple frozen teachers into a unified LoRA-Student through Ensemble Knowledge Transfer, eliminating runtime module selection or generation. A semantic-guided distillation strategy weights teacher contributions by feature-space similarity to mitigate catastrophic forgetting and overfitting. Across three class-order runs, TALON achieves comparable or better mean average accuracy across four FSCIL benchmarks, obtaining 86.68 +/- 1.22% on CUB200, 90.39 +/- 0.27% on CIFAR100, 78.38 +/- 0.94% on ImageNet-R, and 96.34 +/- 0.33% on miniImageNet. TALON uses up to 33x fewer deployment parameters and reduces average inference time per task to 26.7 s, a 41.70% reduction relative to ASP.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Machine Learning

• arXiv preprint · 30 September 2026

  ReGain: Restoring Subject Fidelity in Personalization on Synthetic Images

  Shubhang Bhatnagar, Ishan Bhatnagar, Viraj Shah and 1 other
• arXiv preprint · 1 October 2026

  SmoothOperator: Enhancing Representations for Fine-grained Open-set Recognition via Modulated Label Smoothing

  Thiru Thillai Nadarasar Bahavan, Yu Xia, Sachith Seneviratne and 1 other
• arXiv preprint · 30 September 2026

  Code to Control: Synthesizing Parameterized Reactive Controllers

  Zergham Ahmed, Joshua B. Tenenbaum, Chris Bates and 1 other
• arXiv preprint · 1 October 2026

  Fixing a Model That Learned Worse Cancer Means Lower Risk: Monotonic Constraints in Bladder Cancer Recurrence Prediction

  Saram Abbas, David Thomas, Naeem Soomro and 3 others
• arXiv preprint · 30 September 2026

  Molecular Property Prediction under Structural Shift with Tabular Foundation Models

  Jinmo Lee, Dooho Lee, Minho Jeong and 1 other

All new papers in Machine Learning