arXiv preprint ·
How Much Does Random Rotation Help GPTQ? Sharp Worst-Case Rate Gaps in the High-Rate Regime
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
GPTQ is Babai's nearest-plane algorithm on the lattice of the Cholesky factor of the layer's input second moment $Σ$. In the high-rate, entropy-coded regime it needs $\tfrac12\log_2(\AM/\GM)$ of its Cholesky pivots more bits than WaterSIC, which is within $0.255$ bit of the rate-distortion limit for every $Σ$; random rotations close most of this gap in practice, and Ordentlich and Polyanskiy asked for the worst-case gap between rotated GPTQ and WaterSIC. We show that after a Haar rotation the $k$-th pivot converges almost surely, uniformly in $k\geδn$ and over spectra $ν$ of bounded condition number, to $1/S_ν(-k/n)$, a classical formula from the theory of Lyapunov exponents, and use it to answer the question. Over spectra with condition number $κ$ the worst rotated gap is $\tfrac12\log_2κ-\log_2\lnκ+\tfrac12-\tfrac1{\ln2}+o(1)$ bits, attained only by two-point spectra, just $\tfrac12\log_2\lnκ+O(1)$ bits below the worst case without rotation, so rotation cannot replace waterfilling in general. It nearly can on real layers: rotation cuts the cost of an $η$-fraction of outlier eigenvalues of size $μ$ from about $ημ$ to $η^2μ/2$ to leading order, but cannot remove spectral spread on a logarithmic scale; for non-Gaussian weights it also adds a weight-side rate cost, which we characterize exactly under a standard high-rate assumption. On $96$ Hessians of Qwen3 and Llama-3.1 the rotated gap has median $0.04$ bit and maximum $0.36$ bit, against $2$--$5$ bits in the worst case for the same condition numbers, and the limit matches it to within $0.009$ bit.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Information Theory
Polynomial-Time Minimax Prediction for Stationary Renewal Processes
Achievability of CSS codes performance limits over Pauli channels under separate ML decoding with bounded graphical complexity
Secrecy Sum-Rate Maximization for RSMA-Enabled Pinching-Antenna Systems
A Beam-Switched Sparse Circular Array for Passive Wi-Fi Sensing
MDS Algebraic Geometry Codes of Square-Root Length on Hyperelliptic Curves
Get the few papers worth your time in Information Theory every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.