arXiv preprint ·
Test-Time Confidence Adaptation under RL-Induced Trajectory Shift
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Trajectory-based confidence estimation has shown promising performance for large language models (LLMs) by leveraging rich signals from their reasoning processes. However, we find that such confidence estimators can degrade substantially after agentic reinforcement learning (Agentic RL) as it induces substantial shifts in the reasoning trajectories. More importantly, despite these shifts, we uncover a reliability structure that is largely preserved across Base and RL models: $\textbf{more}$ $\textbf{stable}$ $\textbf{trajectories}$ $\textbf{remain}$ $\textbf{more}$ $\textbf{likely}$ $\textbf{to}$ $\textbf{produce}$ $\textbf{correct}$ $\textbf{answers}$. This preserved relationship provides a transferable, label-free reliability signal for post-RL confidence adaptation. Building on this finding, we propose $\textbf{StaR-TTA}$, a stability-guided framework for label-free test-time confidence adaptation. StaR-TTA jointly learns from correctness supervision and stability-derived reliability supervision on labeled Base trajectories, establishing the connection between trajectory stability and answer reliability. At test time, it reuses stability-induced reliability rankings from unlabeled RL trajectories to adapt confidence scores to the shifted trajectory distribution, without requiring target correctness labels. Across four QA benchmarks, StaR-TTA achieves an average AUROC of \(0.721\), with consistent improvements across unseen datasets, model families and scales, and different Agentic RL algorithms.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Artificial Intelligence
SlowLab: Benchmarking Autonomous AI Agents on a Year of Experiments They Cannot Undo
Repulsive Population Policy Gradients for Behaviorally Diverse Reinforcement Learning
RobotAgentBench: Towards Open-world Robot Autonomy
LM-DSB: A Lightweight Multi-agent System for Automated Daily Safety Bulletin Generation in Construction
Thinking-in-Bets at Test Time Reduces Sycophantic Revisions in LLM Agents
Get the few papers worth your time in Artificial Intelligence every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.