arXiv preprint ·
Teaching Vision-Language Models to Decline: Support Completion Against Mirage
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Vision-language models answer questions about images they were never shown, a failure called mirage. Such an answer can be correct without being justified by the evidence, and the difference matters wherever a model's output is checked by someone who did not see its input. We locate the failure in the training objective: cross-entropy separates a model that reads the image from one that ignores it by only the conditional mutual information between answer and image, and no training example rewards declining. We formalise support completion, TIDE, which also shows each training question with a blank image, a same-domain wrong image and an out-of-domain image, with targets that decline, and prove that under TIDE every image-blind predictor is suboptimal by at least 1.04 nats per example, that the optimum declines exactly on the counterfactual kinds while keeping its behaviour on valid images, and that for any finite model the error of its answer-or-decline decision, declining on valid input plus answering on invalid input, is bounded below by how well its representation separates valid from invalid evidence. On Qwen2.5-VL-7B under a frozen held-out protocol with two judges, the three-condition predecessor of TIDE reduced the invalid-evidence mirage rate from 0.759 to 0.097 and the blank-image rate from 0.700 to 0.003 at a false-abstention cost of 0.126; label-free grounding raised the rate to 0.859, a likelihood-advantage contrast moved it only to 0.718 because its empty-image term carries no gradient, and grounding applied after support completion re-opened the mirage. The four-condition protocol on three base models, with seeds and component ablations, is in progress and not reported here.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Computer Vision
Meaningful Moments: A Corpus of VLM-Derived Temporal Importance Labels for Video Action Recognition
CÆNL: Collapse-Based Active Neural Learning with Objective-Aligned Representation Control
Adversarial Robustness of Camera-Radar Sensor Fusion for Autonomous Driving
Gaussian Splatting for Real-Time Scene Reconstruction: Methods, Systems, and Trade-offs
Removing the Compute Bottleneck for Event-based Visual Place Recognition with Spiking Neural Networks
Get the few papers worth your time in Computer Vision every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.