Internal Stability Signals for Selective Verification and Hallucination Mitigation in Vision-Language Models
Abstract
Large vision-language models (LVLMs) are increasingly used for multimodal search, open-ended question answering, document understanding, and generated content analysis, yet they still produce claims that are fluent but unsupported by visual evidence. Existing hallucination mitigation methods often rely on external multimodal judges, language-model annotators, or human token labels to localize unsupported content, adding cost and introducing a mismatch with the target model's own failure modes. We propose HIS-Guard, a framework for trustworthy multimodal claim generation that diagnoses unsupported claims using the target LVLM's internal stability signals and routes those signals to selective verification or preference optimization. Its core diagnostic, Hallucination Instability Score (HIS), combines semantic convergence from cross-layer LogitLens entropy reduction with visual grounding coherence from cross-token attention-distribution consistency over image patches. By decomposing HIS into semantic and visual instability, HIS-Guard supports claim-level credibility analysis without external token-level hallucination localizers or auxiliary multimodal judges. We then instantiate a Diagnose-Treat-Prevent preference objective (DTP-DPO): semantic instability treats risky tokens by weighting DPO rewards, while visual and combined instability prevent visual neglect through gated auxiliary regularizers. Across Qwen2.5-VL-3B/7B and InternVL2.5-2B, HIS-guided optimization provides downstream evidence that the diagnostic captures useful failure modes. On Qwen2.5-VL-3B, HIS-Guard reduces CHAIR_s by 32.2% and CHAIR_i by 45.6% over the baseline, improves over standard DPO trained on identical data, and largely preserves POPE accuracy. Scaling to Qwen2.5-VL-7B keeps HIS-Guard best on both CHAIR and POPE, while InternVL results suggest that the diagnostic signal is useful but the regularization strength is architecture-sensitive. Together, these results position internal stability as a practical signal for trustworthy multimodal information systems, selective verification, and hallucination-aware evaluation. The public implementation is available at https://github.com/20508Z/HIS-DTP-DPO.