arXiv preprint ·
AdamX: Cosine similarity meets gradient descent
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
We introduce AdamX, a first-order optimizer that incorporates cosine similarity as an adaptive mechanism for controlling update magnitudes. The proposed method is scalable, model-agnostic, and straightforward to integrate into existing training pipelines. We further introduce a variance rectification scheme that promotes smoother optimization during the early stages of training. Overall, we provide empirical evidence that AdamX achieves competitive convergence rates across a range of benchmark datasets and architectures. Performance is evaluated in terms of the number of epochs required to reach predefined performance thresholds under a fixed hyperparameter budget. Code and Experiments available at: https://github.com/FranciscoCaldas/adamX.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Machine Learning
EGGROLL, Unrolled: Understanding and Improving Low-Rank Evolution Strategies at Scale
Inference for Newton Methods with Accelerated Sketch-and-Project via Random Scaling
Conformal-DRO: Distributionally Robust Optimization with Conformalized Ambiguity Set
Linear Exponential Quadratic Gaussian Covariance Steering
Convex Optimization with Nested Evolving Feasible Sets (CONES) under Time-Varying Loss Functions
Get the few papers worth your time in Machine Learning every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.