Arithmetic Simplicity in Stochastic Gradient Methods

Abstract

A gradient descent method is arithmetically simple if the operations are limited to $+,-, \times$, and division $x/2^t$ with integer $t$. An arthmetically simple gradient method is easy to implement in chip design. We show how to transform AdaGrad, Adam, and AdamW into arithmetically simple. AdamW is based on the recursion $x_{t+1}=(1-λη)x_t-\frac{η}{s}m_t$ and Adam is the special case of AdamW with $λ=0$. We transform them into a static case with $s=S(T)$, where $T$ is the number of iterations, and $S(T)$ is a fixed function. The convergence analysis is given for a static Adam, which is also arithmetically simple.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Machine Learning

• arXiv preprint · 26 September 2026

  Emergent One-Third Scaling Law as Attention Tries to Concentrate

  Yizhou Liu, Sara Kangaslahti and Jeff Gore
• arXiv preprint · 27 September 2026

  Byzantine-Robust Federated RAG via Aligned Calibration and Fixed-Membership Conformal Prediction

  Prasanjit Dubey, Aritra Guha and Xiaoming Huo
• arXiv preprint · 26 September 2026

  LLM Unlearning Evaluation with TRIAGE

  Danial Ataee and Peter Triantafillou
• arXiv preprint · 27 September 2026

  Improving the Diversity of LLM Outputs without a Trade-off

  Ryoma Sato
• arXiv preprint · 26 September 2026

  An Attention-Driven Heterogeneous GNN Model for Credit Card Fraud Detection

  Kathiresan Jayabalan and Sethuraman Radhakrishnan

All new papers in Machine Learning