arXiv preprint ·
Tracking the Moving Frontier: Long-Short Term Advantage Estimator
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Group-based RLVR methods estimate advantages by repeatedly sampling multiple trajectories for each prompt, making long-horizon agent training expensive and discarding useful experience accumulated across iterations. We ask whether historical experience can replace these repeated within-iteration comparisons without directly optimizing on stale trajectories. We introduce Long-Short Term Advantage Estimator (LSTAE), a single-stream RL algorithm that uses history for advantage estimation while updating the policy only with the current rollout. LSTAE maintains a persistent tracker for each task anchor. At the trajectory level (long term), a drift-aware historical baseline tracks the anchor's moving success frontier and measures the relative contribution of each new trajectory. At the step level (short term), a recent state-experience buffer exploits recurrent states to estimate localized action advantages. This two-timescale design converts accumulated experience into multi-granular credit signals, requiring only one rollout per anchor. Across agentic and mathematical reasoning benchmarks, LSTAE matches or improves upon strong group-based baselines while substantially reducing rollout cost.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Machine Learning
Linear Algebra Foundations of Efficient Attention: A Phase Reversal in Rank Collapse Under SVD Compression
Noisy-Space Policy Gradient for Diffusion Policies in Offline Reinforcement Learning
AGSA-Net: Abundance-Guided Self-Attention Network for Spectral Unmixing-Aware Hyperspectral Remote Sensing Image Classification
Human-agent discovery of reconfigurable in-plane ferroelectric superdomain control
AutoKD: Autonomous Knowledge Discovery
Get the few papers worth your time in Machine Learning every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.