arXiv preprint ·
Feature Normalization under Finite Training Budgets
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Standardizing a rare feature can help stochastic gradient descent learn its effect within a finite budget, but it also changes the noise cost of learning. We characterize the lowest rare-subgroup risks attainable by standardized and unscaled training when each chooses a nonnegative constant learning rate and a deterministic stopping time under the same observation cap. For two independent binary features, we derive both optimized risks as prevalence vanishes while the expected number of rare observations remains finite. Uniform control over rates and stopping times reduces each limiting oracle to a scalar minimization. With positive noise, the preferred scaling changes exactly once as we move signal to the rare feature while the limiting risk at initialization stays fixed and positive. The formulas also explain how noise and budget affect the comparison. Separate finite-prevalence bounds favor standardization on a rare-dominant signal family and unscaled training on a common-dominant family. The results concern final iterates with population preprocessing and fresh observations. Exact-moment searches assess the approximation at finite prevalence. In a controlled study, both orderings persist under observation reuse, estimated preprocessing, and intercept fitting when tuning choices are fixed before evaluation. Selecting a predictor after observing training can reverse an ordering, even when selection uses exact population risk.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Machine Learning (Statistics)
A Mean Field Model of Consumption Inertia and Keeping Up with the Joneses
Convex-Analytic View of Query Selection for Safety Verification of Markov Decision Processes
Online Submodular Welfare Maximization: Computation is not the Bottleneck
Risk-Averse Inference-Time Diffusion Model Alignment with Finite-Sample Guarantees
Exploration strategies for Gaussian process bandit optimization with inference regret guarantees
Get the few papers worth your time in Machine Learning (Statistics) every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.