Data-Efficient Classifier Pretraining Through Diffusion Models

Abstract

Machine learning with limited data is susceptible to overfitting, especially with modern deep neural networks. Prior to the introduction of AlexNet, reconstruction-based pretraining with auto-encoder and denoising auto-encoder were found to be beneficial, partially due to the limited size of the training sets at the time. Motivated by these traditional approaches, we propose DiffPT, a simple diffusion modeling approach to pretrain a given classifier backbone for limited domain-specific training data. We experiment on time-series data, standard natural-image benchmarks, and medical imaging datasets, and contrast the diffusion approach to a multitude of modern and traditional un/self-supervised methods. While contrastive pretraining performs better on CIFAR-10 and Tiny ImageNet in low-data regimes, DiffPT is particularly effective on medical imaging and time-series tasks, where it outperforms self-supervised baselines across a range of low-data regimes. Our results suggest that DiffPT is a simple and effective approach for data-efficient classifier pretraining on different modalities.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Machine Learning

• arXiv preprint · 16 October 2026

  Shape-Aware Neural Time Fields for Rigid-Body Motion and Manipulation Planning

  Yanming Zhou, Yuchen Liu and Ahmed H. Qureshi
• arXiv preprint · 17 October 2026

  Power-Law Correlations Without Apparent Temperature-Tuned Criticality in Fixed-Tree Hierarchical Languages

  Marcel Kühn and Bernd Rosenow
• arXiv preprint · 16 October 2026

  Acquisition Function-Guided Generation for Bayesian Optimization over Discrete Spaces

  Tommaso Ancilli, Thomas Wedenig, Henry Moss and 1 other
• arXiv preprint · 17 October 2026

  Uniform convergence at optimal rate and scale

  Noga Alon, Steve Hanneke, Shay Moran and 1 other
• arXiv preprint · 16 October 2026

  One Communicator, Three Vendors: Extending the MoA Predictive Theory to a Real NVIDIA+AMD+CPU MPI Ensemble

  Lenore Mullin and Gaetan Hains

All new papers in Machine Learning