Localized Conformal Safety Monitoring with Vision-Language Models for Autonomous Driving

Abstract

Monitoring planned driving trajectories requires accurately estimating the collision likelihood with actors whose motion is itself impacted by the ego motion. Existing classical approaches are often limited by the quality of their forecasting model. Vision-language models (VLMs) have shown promise in reasoning about the consequences of high-level actions, yet their approximate predictions are unsuitable for safety-critical applications such as autonomous driving. Conformal prediction (CP) has emerged as a data-driven framework for quantifying the uncertainty of black-box model predictions. We propose Split Label-Localized Conformal Prediction (SLLCP), a post-hoc calibration layer over frozen VLMs that transforms their unreliable predictions into probabilistically calibrated safety prediction sets. We consider how the ability to estimate safety can depend on the observed driving scene and introduce a localized procedure that upweights relevant past experience when calculating uncertainty thresholds. We provide label-conditional finite-sample distribution-free coverage under exchangeability. Evaluated over 15k CARLA trajectories from unseen scenarios, SLLCP correctly flags 89.6% of collision-causing trajectories with a Qwen backbone and 88.4% with a Cosmos backbone, while the base VLMs only flagged 4.6% and 39.1% of the collision-causing trajectories, respectively. These results indicate that local, label-conditional calibration can reduce missed unsafe trajectories.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Robotics

• arXiv preprint · 2 October 2026

  Mind the Refinement Gap: When Safe High-Level Robot Plans Produce Unsafe Executions

  Stabak Das, Priyesh Ranjan, Xiangfang Li and 1 other
• arXiv preprint · 3 October 2026

  Autoware in Construction: Gap Analysis and LiDAR Perception Toward Off-Road Autonomous Driving

  Yu Otsuki, Teja Emmey, Sena Matsushita and 2 others
• arXiv preprint · 2 October 2026

  A Passive AI System for Verifying Physical State on Automated Liquid Handlers

  Junqiong Joanne Qiu, Zeckria Kamrany, Ananya Anand and 7 others
• arXiv preprint · 3 October 2026

  CurveCodec 2: Skeleton-agnostic animation compression with a learned entropy model

  Mingyi Shi, Huancheng Lin, Xuelin Chen and 1 other
• arXiv preprint · 2 October 2026

  DeltaWorld: Physically Consistent Interactive World Simulators via Action-Conditioned Latent Increment Learning

  Boyuan Hou, Xiaoge Cao, Chaofan Zhang and 2 others

All new papers in Robotics