Is Success All You Need? Investigating the Impact of Input Perturbations on VLA Behaviour in Tabletop Manipulation Tasks

Abstract

Vision-Language-Action (VLA) models have achieved high task success rates on robot manipulation task benchmarks. More recently, there has been an emphasis on evaluating the robustness of VLA models to perturbations. However, this robustness is still predominantly measured through Task Success Rate (TSR). In this work, we propose a benchmark-agnostic evaluation framework to measure the behavioural robustness of models by characterising how successful trajectories are executed under perturbation. We implement this methodology by extending the widely-used LIBERO and LIBERO-Plus benchmarks. Across three state-of-the-art VLA models, four LIBERO task suites and seven perturbation conditions, we evaluate changes in both typical successful behaviour and its variability, including metrics of motion smoothness, efficiency and gripper behaviour. We find that perturbations can alter the behaviour of successful trajectories, a phenomenon which cannot necessarily be inferred from TSR alone. Across LIBERO suites, we identify cases where state-of-the-art VLA models achieve comparable TSR under the same perturbation condition, yet behaviour on successful trajectories diverges substantially. Therefore, to have a more robust assessment of task performance, we argue that suitable measures of robustness should capture not only whether a task is completed, but also how the robot behaves while completing it. When evaluating the robustness of VLA models, TSR may be complemented by behavioural evaluation metrics that characterise the nature and variability of successful task execution by robots.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Robotics

• arXiv preprint · 1 October 2026

  Are Frontier VLM Agents Ready to Be Robot Generalists? An Empirical Study with the Embodied Agent Arena

  Haojian Huang, Pukun Zhao, Zexi Li and 7 others
• arXiv preprint · 2 October 2026

  Mind the Refinement Gap: When Safe High-Level Robot Plans Produce Unsafe Executions

  Stabak Das, Priyesh Ranjan, Xiangfang Li and 1 other
• arXiv preprint · 1 October 2026

  Lang3DSeg: Annotation-Free Open-Vocabulary 3D Segmentation with Point Transformers

  Cigdem Kokenoz, Amir Salarpour, Alkim Domeke and 5 others
• arXiv preprint · 2 October 2026

  A Passive AI System for Verifying Physical State on Automated Liquid Handlers

  Junqiong Joanne Qiu, Zeckria Kamrany, Ananya Anand and 7 others
• arXiv preprint · 1 October 2026

  Kinematic MeanFlow: One-Step Action Generation Policy for Robotic Foundation Models

  Jiawei Fan, Sifeng Wang, Yuqing Hou and 1 other

All new papers in Robotics