Beyond Visual Enhancement: Adaptive Multi-Context Steering to Mitigate LVLM Hallucinations

Abstract

Hallucination remains a significant challenge in Large Vision-Language Models (LVLMs). Existing training-free methods generally mitigate hallucinations through contrastive decoding or visual enhancement, often increasing the relative influence of visual evidence during generation. This raises a fundamental question: Can LVLMs dynamically regulate the contributions of different context sources to suppress hallucinations? In this work, we investigate and quantify how LVLMs coordinate multiple context sources during decoding and examine how this intrinsic behavior can guide hallucination mitigation. We find that LVLMs exhibit an intrinsic vision-attending tendency that can guide adaptive visual steering, while textual contexts can also contribute to hallucination mitigation. Motivated by these findings, we propose AIMS (Adaptive Information Multi-source Steering), a lightweight training-free framework that adaptively coordinates visual, prefilled textual, and generated contexts during decoding. Specifically, AIMS constructs compact prototypes for the three context domains and estimates their affinities with the current query to determine head-wise steering weights. The resulting multi-source steering direction is applied to the query representation, enabling adaptive context integration without additional model training or auxiliary forward passes. Extensive experiments across multiple LVLMs and decoding strategies demonstrate that AIMS effectively mitigates object hallucination while maintaining competitive general-purpose multimodal capabilities.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Computer Vision

• arXiv preprint · 8 October 2026

  Mid-Training Language Models on Raw Video

  Jaedong Hwang, Xiaoqian Shen, Ernie Chang and 9 others
• arXiv preprint · 9 October 2026

  Structure-Aware Matching for Robust UAV Thermal-to-Satellite Geo-Localization

  Chun Liu, Yuxin Wang, Zhenxiang Huo and 3 others
• arXiv preprint · 8 October 2026

  Expression-Diverse References for Identity-Preserving Video Generation

  Tianwen Fu, Wenbin Teng, Gonglin Chen and 3 others
• arXiv preprint · 9 October 2026

  APEX: Asynchronous Prefix Denoising for Extreme Compression

  David Dinkevich, Nisan Chiprut, Yoav HaCohen and 1 other
• arXiv preprint · 8 October 2026

  Transforming Image Editors into Video Editors

  Feng Wang, Zijie Li, Ceyuan Yang and 2 others

All new papers in Computer Vision