arXiv preprint ·
Who Sets the Pace of Judge Hacking? Judge Capability versus the Policy-Judge Gap
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Under GRPO against a frozen, reference-free LLM judge, RL exploits the judge's false positives. Does the policy-judge capability gap or the judge itself set how fast and how far? We train 0.5B and 1.5B Qwen2.5 policies on GSM8K against four judges at least as large, and the 1.5B policy also against one smaller judge. In a post-hoc analysis, judge acceptance rose, but no cell of 12 gained more than 0.04 in seed-mean held-out greedy accuracy. A gold-reward control with the same recipe raised the 1.5B policy's accuracy by +0.054 (for the 0.5B policy, gold reward gave no detectable gain either). The depth of judge hacking tracks the judge: the plateau correlates with judge AUC (Spearman -0.69 over 12 cells), and within a family the larger judge is fooled less deeply. For the pace, two pre-specified pairs contradict the gap account: the cell with the larger gap was fooled 3.5x sooner at the pre-specified threshold. With 12 cells, however, we cannot formally separate judge AUC from the gap for either endpoint. A blind test of step-0 predictors on three held-out judges was inconclusive. At 7-8B, judges from three families plateaued between 0.37 and 0.85.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Machine Learning
Sequent-Prover: Training Agents for Formal, Checkable SMT-based Reasoning
PAIR: Memory-Efficient and Fast Inference with Reading--Reasoning Fusion
Do correct answers have stable latent knowledge support?
Improving the Evaluation of Biomedical Question Answering with Few Human Preference Signals
North or South? Diagnosing LLMs' Understanding of North Korean Language, Society, and Knowledge
Get the few papers worth your time in Machine Learning every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.