arXiv preprint ·
FidelityQ: Preserving Predictive Fidelity in Ultra-Low-Bit Language Model Quantization
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Ultra-low-bit scalar quantization offers a regular arithmetic structure for efficient deployment of large language models, but preserving predictive behavior at extreme precision remains challenging: with only a few reconstruction levels, accurate weight representation must translate into consistent predictions from a small calibration set. We propose FidelityQ, a scalar quantization framework that preserves predictive fidelity at weight precisions down to 2 bits. For accurate quantization, we introduce Information-Aligned Analytic Coding (IAC), which shapes each weight row toward a centered symmetric source via randomized Hadamard mixing and fits a mirrored four-level minimum-distortion quantizer to its empirical weights. Every level set in this family decomposes exactly into two sign-valued bases with associated scales, combining source-adaptive reconstruction with lookup-free decoding compatible with scaled integer arithmetic. Our analysis further shows that residual predictive discrepancies are heavy-tailed and concentrate on tokens the teacher predicts confidently, motivating Knowledge-Intensive Calibration (KIC) for consistent quantization. KIC ranks sequences by the tail of token-wise teacher-student divergence, allocates a limited calibration budget to the highest-scoring samples, and restores predictive agreement by updating only quantization levels and scales under teacher supervision, keeping pretrained weights frozen. Experiments on the LLaMA and Qwen families demonstrate the effectiveness of FidelityQ in weight-only and joint quantization settings. On LLaMA-2-7B, FidelityQ achieves a WikiText2 perplexity of 6.62 at W2A16 and improves the prior state-of-the-art W2A4KV4 result from 8.31 to 6.96, while reducing linear-layer weight memory by 8.0x relative to FP16.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Computation & Language / NLP
Scaling Laws for Multimodal Data Mixtures
Sequent-Prover: Training Agents for Formal, Checkable SMT-based Reasoning
MemForm: Benchmarking Memory Formation Poisoning in LLM Agents
Do correct answers have stable latent knowledge support?
Neither True nor False: Do LLMs Represent Truth-Value Gaps in Plural Predication?
Get the few papers worth your time in Computation & Language / NLP every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.