RACE: Calibrated Cumulative Scoring for Rollout-Free Robot Policy Checkpoint Selection

Abstract

Assessing the deployment performance of learned robot policies during training is challenging, as directly measuring task success requires costly real-world trials for each candidate checkpoint. Averaged validation loss offers a rollout-free alternative, but its checkpoint rankings can be inconsistent with deployment performance. We analyze two limitations of this criterion: it penalizes action deviations without considering whether the policy can recover from them, and gives later reference predictions equal weight even when earlier deviations may lead execution away from the reference. We propose RACE, a novel rollout-free metric for ranking checkpoints using a fixed set of held-out successful trajectories. RACE calibrates action deviations against state-conditional thresholds derived from action differences at similar states in these trajectories. We aggregate these scores through a running product, so that a substantial deviation reduces the contribution of subsequent reference predictions. Averaging the resulting cumulative scores over time and across trajectories yields the checkpoint score. In simulation, RACE outperforms baselines across validation sources, while several baselines degrade sharply when the validation set changes. On real robots, its rankings align with deployment success across the evaluated policy backbones and tasks of varying difficulty. Perturbation analyses further show that RACE distinguishes deviations with markedly different effects on task success, even when baselines respond similarly.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Robotics

• arXiv preprint · 4 November 2026

  Closed-Loop Adaptive Sampling on a Generative Ocean Model

  Shayesteh Hafezi and Alan Papalia
• arXiv preprint · 5 November 2026

  An Empirical Study of Reinforcement Learning for Keypoint Positioning of Deformable Linear Objects

  Vinay Rodrigues, Sara Zeynalpour, Gokhan Solak and 2 others
• arXiv preprint · 4 November 2026

  ZipDrift: Generating Multimodal Short-Horizon MPC Costs with One-Step Drifting Models

  Ke Zhang
• arXiv preprint · 5 November 2026

  CRISP: Constraint-Resolved Interactions between Subsystems for Predictive Whole-Body Cooperation

  Dario Valenzuela and Jeeseop Kim
• arXiv preprint · 4 November 2026

  WayPOP: A Panoramic Open-Set Panoptic Tracking Benchmark

  Rohit Mohan, Maximilian Luz, Swarnava Chowdhury and 7 others

All new papers in Robotics