arXiv preprint ·
Sparse Regression Distilled from a Single Robust Fit
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Robust linear fits can resist response contamination yet remain too dense or unstable for useful global explanations. We propose penalized distillation, which fits a smoothly clipped absolute deviation (SCAD) estimator to a robust initial estimator's empirical fitted surface along a safeguarded coordinate-descent path and evaluates candidate states separately for fidelity, parsimony, perturbation stability, and held-out prediction. The new results attach to the states the algorithm actually computes. Conditional on a fixed uncontaminated design, deterministic bounds transfer response-replacement boundedness from the initial fit to every retained path state. Turning to fixed dimension, we characterize the oracle-support branch by its empirical-Gram projection and influence function, give conditions for covariance-weighted least-squares approximation equivalence, and establish a path-conditional generalized information criterion. By contrast, at large dimension-to-sample ratios the full-coordinate robust fit collapses without warning, and screening restores the construction. Under a sure-screening framework, the robustness bound and the support and selection guarantees transfer to the screened fit. Simulations separate robustness transfer from support recovery, efficiency, and computation across the dimension-to-sample ratio, with p up to 240, and the signal density, which isolates what the sparse stage adds once the screen over-selects. In a duplicate-grouped superconductivity study, the distilled estimator remains predictively stable under prespecified training-response shifts but retains 66.8--68.8 of 81 slopes. Stronger sparsification reduces the model to 12.6--14.0 slopes only at visible fidelity and prediction cost. Distillation therefore preserves predictive stability on these data without substantiating a compact coordinate-level explanation.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Machine Learning (Statistics)
Distributional Balancing with Machine Learning for Clinical Trial Augmentation Using Real-World Data
Doubly robust target inference for generalized linear regression with completely missing covariates
PACE: Plug-and-Play Contextual Embedding for Feature Screening with Pretrained Tabular Foundation Models
Model-Agnostic Feature Selection via LOCO-Guided Adaptive Minipatch Sampling
Belted Engression: Sufficient Dimension Reduction for Generative Distributional Regression
Get the few papers worth your time in Machine Learning (Statistics) every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.