SADI: Spatial-Aware Dynamic Intervention for Mitigating Hallucinations in LVLMs

Abstract

Large Vision-Language Models (LVLMs) persistently suffer from object hallucinations, generating plausible yet factually incorrect outputs that fabricate objects non-existent in the visual input. Existing mitigation methods typically fall into three paradigms: fine-tuning with targeted data, which incurs prohibitive computational costs; applying contrastive decoding, which introduces high inference latency and may penalize beneficial language priors; or intervening in attention weights via heuristic hard-truncation or head-homogeneous shifts, operations that may discard information from selected attention paths. Building upon the third paradigm, we present Spatial-Aware Dynamic Intervention (SADI). SADI discards conventional rigid truncations in favor of a spatial-aware soft mask to adaptively correct visual-semantic misalignments. Specifically, it establishes a robust median consensus among multiple attention heads to anchor reliable visual representations. Guided by the spatial variance across these heads, SADI dynamically allocates intervention weights and employs a soft mask to gently suppress anomalous attention distributions. As a lightweight, training-free inference solution, our method reframes attention modulation as a smooth, continuous recalibration process. Ultimately, SADI effectively rectifies semantic drift while maintaining comparable generation fluency and description richness in our evaluations. Extensive experiments demonstrate that SADI significantly reduces both sentence- and instance-level hallucinations while maintaining description richness. Across the evaluated CHAIR, POPE, and MME benchmarks, SADI shows consistent gains in the tested settings. Code is released at https://github.com/yz-8/SADI.