arXiv preprint ·
Revisiting Error Feedback for Model Quantization
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
We study causal error-feedback coefficients as design variables for model quantization. Modeling instantaneous quantization errors as independent, zero-mean variables with a common fixed variance, we formulated a metric-dependent trace problem for designing feedback coefficients. The formulation resembles fixed-order GPTQ/LDLQ feedback and supports Jacobian-aware designs that account for downstream error propagation. For a positive-definite metric $\boldsymbol{H}$, we prove that the joint rotation-feedback trace minimum is $n\det(\boldsymbol{H})^{1/n}$ and present an attaining pair using the geometric mean decomposition. Experiments on synthetic networks, trained MLPs, and language models show benefits from downstream-sensitive feedback, but also reveal that trace optimality need not improve finite-level performance. Small QAT experiments further illustrate direct loss-driven feedback tuning with fixed or trainable network weights.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Machine Learning
FedLoRD: Personalized Federated Learning based on Low-Rank Structures of Model Parameter Differences
Train to Stitch: Topology-Preserving Regularization for Interoperable Representations
ECG-RSI: Recursive Self-Improvement through Targeted Electrocardiogram Waveform Evolution
ONCship: A Large-Scale Underwater Acoustic Corpus for Ship Recognition
DeepPV-MRF: End-to-End Partial-Volume Fraction Estimation from MR Fingerprinting Time Series
Get the few papers worth your time in Machine Learning every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.