arXiv preprint ·
RBGA: Residual-Budget Guide Arbitration for Offline-to-Online Safe Reinforcement Learning in Dexterous Manipulation
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Dexterous manipulation requires robots to leverage existing datasets for rapid skill acquisition while maintaining safety during real-world deployment to prevent damage to the robot or its surroundings. Offline-to-online safe reinforcement learning (O2O safe RL) provides a promising solution for this problem by leveraging prior experience for online improvement under safety constraints, thereby mitigating the sample inefficiency and exploration risk of learning safe policies from scratch. However, existing O2O safe RL approaches do not explicitly account for the safety budget already consumed when making decisions within an episode. Excessive cost incurred early can therefore leave little room for subsequent exploration and policy improvement. To address this limitation, we propose \textbf{R}esidual-\textbf{B}udget \textbf{G}uide \textbf{A}rbitration (RBGA), an algorithm that explicitly tracks the remaining episode safety budget and uses it to adaptively select between offline guidance and the evolving online policy. At each step, RBGA compares the cost-to-go estimates of the guide and online actions against the remaining budget, enabling step-wise action arbitration throughout the episode. We provide theoretical analysis characterizing the safety implications of residual-budget-based selection and its effect on guide-assisted exploration. Extensive experiments demonstrate that RBGA enables effective online policy improvement while maintaining strong safety performance. Our code is available at this \href{https://github.com/sii-pilab}{link}.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Machine Learning
One Call Ahead: Packed Self-Speculative Decoding with Diffusion Drafting
Continual Learning Exam: Benchmarking Long-Horizon RSI and Experiential Learning in LLM Agents
The Cost of an Unknown Horizon for the Last Iterate of Clipped SGD under Heavy-Tailed Noise
A Nonconvex Framework for Joint Sparse Sample Weighting and Parameter Estimation
Faithful Federated Tabular Data Synthesis
Get the few papers worth your time in Machine Learning every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.