arXiv preprint ·
Training-Free Prompt Selection for Efficient Reinforcement Learning With Partial Rollouts
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Reinforcement learning with verifiable rewards (RLVR) has emerged as a prominent paradigm for training reasoning models, but its computational cost makes efficient data selection increasingly important. Since training examples contribute unequally to learning, selecting a small, informative subset can offer a more efficient alternative to using the entire data pool. The training-free selection method estimates prompt value with hidden states from complete greedy rollouts. However, in our datasets, $26$--$32\%$ of these rollouts fail to terminate within the generation budget. We introduce LIFT, a training-free data selector that identifies impactful prompts using only partial reasoning rollouts. At its core, LIFT uses Partial Rollout Context Lift (PRCL), a score inspired by pointwise $\mathcal{V}$-information that accumulates token-level log-likelihood ratios between a question-conditioned context and a null context with the question removed. This enables prompt scoring from a short rollout prefix rather than a complete trajectory. Across Qwen2.5-Math-1.5B and DeepSeek-R1-Distill-1.5B, prefixes retaining at least 90\% and 95\% of the full-rollout PRCL score, respectively, require only 26\% and 4.7\% of the tokens generated by full rollouts. Compared with SHIFT, LIFT reduces selection cost by 3.7--9.1$\times$ while achieving competitive downstream RLVR performance, with the highest six-benchmark average in three of six settings.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Artificial Intelligence
Between Spoilers and Guesswork: Avoiding Over-Specification in Coding Benchmarks
Screening LLM Judges Without Ground Truth: A Preliminary Evaluation of an AI Dental Hygiene Coach
StoBasket: Joint Upsell and Cross-Sell Optimization for Contribution Margin
What Matters for NVFP4 Training? A Comparative Study of Low-Precision Pretraining Recipes
When Can You Trust an Agent's Own Verifier? Stress-Testing Counterfactual Rationale Verification Under Search Pressure and Oracle Access
Get the few papers worth your time in Artificial Intelligence every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.