arXiv preprint ·
Rubrics Beyond Evaluation: Exploiting Rubric Structure in Reinforcement Learning
- Published
- Source
- arXiv preprint, posted before peer review
- Published in
- arXiv
- DOI
- 10.48550/arxiv.2610.25850
- Original
- Read it at arXiv
- Cited by
- 0 works, as OpenAlex counts them
- Topics
Abstract
Rubrics are widely used to evaluate open-ended question answering by breaking overall quality into explicit criteria. These explicit criteria make rubrics a rich source of learning guidance, yet their potential for guiding the training process has been largely overlooked. Existing methods use rubrics mainly as direct feedback signals, without explicitly modeling the capabilities behind their criteria. This leaves their capability structure largely unused in guiding the training process. We introduce Rubric-Structured Reinforcement Learning (RSRL), an RL framework that unlocks the full potential of rubrics by exploiting their structure throughout the training process, rather than treating them simply as direct reward signals. At its core, RSRL builds on three key designs: (1) Rubric-to-Capability Structuring first recovers the capability structure hidden in rubric criteria, (2) Self-Sufficient Capability Learning then uses this structure to bootstrap underdeveloped capabilities into effective learning, (3) Capability-Residual Credit Routing finally preserves capability-specific gains during optimization. Experiments show that RSRL consistently improves both overall performance and underdeveloped capabilities, boosting downstream scientific question-answering performance by up to 7.43 over the initial policy and 7.60 over non-rubric structured RL baselines, showing that reinforcement learning can substantially benefit from the capability structure encoded in rubrics.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Machine Learning
When Better Opportunities Backfire: Service Worker Responses to Parallel AI Deployment in Customer Acquisition
KPHarness: Co-Evolving Knowledge and Probing for Long-Horizon Gaming Harnesses
Safety Offset Learning: When Benign Fine-Tuning Misaligns LLMs
The Router Already Knows
Weighted State Souping for Efficient Long-Context Retrieval
Get the few papers worth your time in Machine Learning every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.