arXiv preprint ·
Understanding Practical Optimization in Machine Learning
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
In brief
Theory That Matches the Training Loop
The thesis analyzes several practical optimization methods while retaining the details that shape their behavior. It identifies when submodel training and adaptive clipping introduce bias, provides convergence results for adjusted methods, and studies layer-wise optimizers including Muon- and Scion-style approaches.
What they found: The thesis finds that Independent Subnetwork Training can converge to a shifted fixed point because sketching before differentiation biases its update for the original objective; Model-Agnostic Sparsified Training avoids this issue by building the sketch distribution into its objective. For private optimization, fixed-quantile adaptive clipping remains biased, while suitable schedules recover near-optimal nonconvex rates under heavy-tailed noise.
AI-written summary. Check anything you rely on against the original paper.
Abstract
Modern machine learning relies on practical optimization methods that often differ from the algorithms studied in classical analyses: they train submodels, clip or normalize gradients, use layer-wise non-Euclidean directions, or approximate costly inner computations. This thesis analyzes the objective, assumptions, update, and computation defined by these procedures, rather than those of simplified proxies. For submodel training, we show that Independent Subnetwork Training can converge to a shifted fixed point because sketching before differentiation biases the update for the original objective. Model-Agnostic Sparsified Training instead incorporates the sketch distribution into its objective, which makes sketched gradients unbiased and admits convergence guarantees. For private optimization, we show that adaptive quantile clipping retains bias at any fixed quantile, while suitable quantile schedules recover near-optimal nonconvex rates under heavy-tailed noise. We then combine smoothed normalization with error compensation and obtain distributed private convergence guarantees under standard assumptions, without assuming bounded gradients. For optimizers based on linear minimization oracles, we develop and validate a layer-wise generalized smoothness model and build on it the Gluon framework for Muon- and Scion-style methods. We also show that the approximate orthogonalization in Muon matters: less precise orthogonalization calls for smaller stepsizes, so its precision should be tuned together with the learning rate. Overall, practical methods are better understood when theory retains the features that determine their optimization behavior.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Machine Learning
Rank-one Factored Gradient Descent without Restricted Isometry: Application to Quantum Pure State Tomography on Multi-Qubits
NextEval: An Agent-Based Solver and Benchmark for Derivative-Free Optimization
Reinforcement learning for constrained stochastic linear--quadratic control in continuous time
QESChunker: A Single Objective Unifies Overlapping and Non-Overlapping Chunking for RAG
Global $\mathcal{O}\left(k^{-2}\right)$ Convergence of Multilevel Gradient-Regularized Newton Methods
Get the few papers worth your time in Machine Learning every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.