arXiv preprint ·
Seeing is Believing: Benchmarking and Mitigating the Interference of Language Priors on Visual Perception in MLLMs
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Language priors in MLLMs can override visual evidence, impairing visual perception. Existing benchmarks assume shared priors across models and rely solely on accuracy. However, cross-model prior agreement falls below 80%, and over 15% of items can be answered correctly without any image, causing accuracy to overestimate prior resistance. To address these limitations, we introduce ViSiP-Bench, a benchmark grounded in rare-but-real scenarios across six prior categories, with a model-aware dynamic prior induction protocol and three metrics that more precisely and continuously measure prior resistance than accuracy alone. Evaluating 24 models reveals that prior adherence is pervasive, does not improve with scale, and larger models tend to either strongly override their priors or firmly adhere to them. We therefore propose MAPS, which identifies each model's strongest priors and fine-tunes on factually grounded knowledge to soften excessive prior reliance without disrupting general capabilities.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Computer Vision
RoPE-Flow: Monocular Vision-based Robot Pose Estimation via 2D Graph-Conditioned 3D Flow Matching
Direct Conditional Parameterization for N-Dimensional Splatting
SIEVE: Overcoming Topological Obstruction in Equivariant Self-Supervised Learning
STRIDE: Evaluating Spatiotemporal Reasoning in Driving Edge Cases
RoboCloth: A Large-Scale Real Cloth Material Dataset for Neural Reflectance Reconstruction
Get the few papers worth your time in Computer Vision every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.