Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning

Abstract

Reinforcement learning with verifiable rewards has become a scalable post-training paradigm for large language models. However, training agents over long-horizon interactions presents two challenges: using only terminal rewards makes turn-level credit assignment sparse and high-variance, while sequential mini-batch updates on a fixed rollout batch introduce policy staleness as the current policy drifts from the behavior policy. Existing approaches obtain denser credit via learned critics, process reward models, or on-policy distillation, but they rely on a value network, step-level annotations, or a stronger external teacher that increase training complexity. Meanwhile, importance sampling and clipping mitigate policy drift, but they do not specify a finite target that stops sequential updates from overshooting. In this work, we introduce TreeOPSD, a critic-free, tree-aware posterior self-distillation framework for long-horizon agentic reinforcement learning. Treating each interaction turn as a tree node, TreeOPSD branches alternative continuations from shared prefixes and recursively backs up terminal rewards into implicit turn-level credit. It then maps each credit to a closed-form, bounded teacher distribution parameterized by the likelihood ratio relative to the frozen behavior policy. Minimizing the divergence between this teacher and the student policy recovers the policy-gradient direction as the tangent at the behavior policy while bounding updates toward a finite target across sequential mini-batches. Empirically, TreeOPSD improves agentic capabilities across several competitive benchmarks and model scales, especially for long-horizon tasks, e.g., raising the average accuracy of Qwen3-30B-A3B-Thinking-2507 by 4.97 points over GRPO on the Deep Research setting.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Machine Learning

• arXiv preprint · 16 October 2026

  Self-Evolving Coding Agents via Harness-Policy Optimization

  Zhepei Wei, Wenlin Yao, Baolin Peng and 5 others
• arXiv preprint · 17 October 2026

  What Matters for NVFP4 Training? A Comparative Study of Low-Precision Pretraining Recipes

  Anjulie Agrusa, Andrei Panferov, Elizabeth Wei and 7 others
• arXiv preprint · 16 October 2026

  Prune the Leaky Neurons: Mitigating Private Data Leakage in Fine-Tuned LLMs via Joint-Embedding-Guided Pruning

  Mohamed Shaaban and Mohamed Elmahallawy
• arXiv preprint · 17 October 2026

  FlashMap: Flow Map Speculative Decoding

  Jeff Shen, Zhejian Zhou, Ryan Lee and 2 others
• arXiv preprint · 16 October 2026

  Causal Latent Workspaces for Looped Transformers

  Ilia Zaznov

All new papers in Machine Learning