arXiv preprint ·
When Better Policies Leave Harder Failures: Coverage Shift in VLM Success Monitoring
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Can a better robot policy become harder to monitor? We study a fixed vision-language model (VLM) that judges success while a policy improves. In a controlled 20-seed tray-placement experiment, training selects only simulator-verified successes. Success rises from 66.8% to 97.0%, and accepted hover failures fall from 5.0% to 0.9% of executions. Yet the judge accepts 50.0% of remaining hovers, versus 31.1% under untrained sampling. These residual failures more often lie below the tray rim, which can conceal the gap beneath a grasped cube. A separate boundary probe accepts 148/160 below-rim hovers and 0/160 above-rim hovers. We call this change in conditional monitoring sensitivity coverage shift: learning changes which failures reach a fixed judge. A broader diagnosis covers nine judges and 1,392 stimuli. Judge-filtered experiments across eight settings further show heterogeneous effects of state-dependent acceptance. The results distinguish task improvement from failure detectability and motivate reporting absolute misses, conditional misses, and audit coverage together.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Robotics
Gen2Dex: Learning Dexterous Manipulation from Simulation-Initialized Generated Video Demonstrations
Dream What I Can: Sparse Visual Planning for Long-Horizon Robot Manipulation
Tension Tells the Tail: Cable-Obstacle Interaction Sensing for Tethered Mobile Robot Navigation
Scalable, Intuitive, Mixed Reality-Based Semi-Autonomous Robotic Telemanipulation for the Execution of a Fine-Grained Bimanual Task
Touch Where It Matters: Learning Local Tactile Policies for Fine-Grained Gripper Control
Get the few papers worth your time in Robotics every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.