On the Comparison of Optimizers for Imbalanced Learning

Abstract

Data imbalance is pervasive in machine learning, from rare words and anomalies to underrepresented patterns in heterogeneous or cross-tabulated data. We study idealized optimizers geometries in continuous time to model small-step training in deep learning. We assume that the source of imbalance is unobserved: the optimizer has only access to the aggregate training loss ignoring the exact contributions of the majority and minority groups. In this setting, we characterize a region where majority losses are optimized regardless of the admissible minority structure. We derive explicit equations of this zone and bounds on the time needed to leave it. These bounds exhibit a milder dependence on minority amplitude for sign, spectral, and Newton descent than for Euclidean gradient descent. Experiments with AdamW and Muon suggest similar advantages over SGD across language, tabular, and image tasks.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Machine Learning (Statistics)

• arXiv preprint · 5 October 2026

  Two-Sample Testing via Path-based Inference

  Eshant English, Wei-Cheng Lai, Yanfeng Yang and 3 others
• arXiv preprint · 6 October 2026

  Is $\sqrt{d}$ Separation Necessary for Gradient EM to Learn Gaussian Mixtures in High Dimensions?

  Yiran Zhang, Mo Zhou, Weihang Xu and 2 others
• arXiv preprint · 5 October 2026

  From Pixels, Without Pre-training: Joint Generative and Self-Supervised Representation Learning in One Model

  Vicente Balmaseda, Ching-Long Lin and Tianbao Yang
• arXiv preprint · 6 October 2026

  Learning a Mixture of GFlowNets

  Tiago da Silva, Amauri H. Souza and Salem Lahlou
• arXiv preprint · 5 October 2026

  Retrieval-Based In-Context Learning: A Domain Adaptation Framework

  Yilun Zhu, Naihao Deng, Yingcong Li and 2 others

All new papers in Machine Learning (Statistics)