Diagnosing Visual Ignorance in Vision-Language Models
TL;DR
Across three VLMs and twelve benchmarks, this work tracks the answer-relevant information decodable at each decoder layer and measures when it becomes coupled to the final output, and distinguishes the availability of visual information from its influence on prediction.
Abstract
Vision-language models (VLMs) achieve high accuracy on many visual question-answering benchmarks, yet it remains unclear whether this accuracy reflects reliable use of the visual details a question depends on. We examine this tension through the lens of visual ignorance: cases where a model can represent task-relevant visual evidence without using it to determine the answer. Across three VLMs and twelve benchmarks, we track the answer-relevant information decodable at each decoder layer and measure when it becomes coupled to the final output. The visual evidence needed for the correct answer can already be read out from intermediate layers, yet it has little effect on the output until the final layers of the decoder. Causal residual interchange confirms that shifting the answer requires the late states, not the earlier ones. We then evaluate all twelve benchmarks under progressive Gaussian blur and find that substantial fractions of predictions survive the entire degradation sequence while continuing to receive benchmark credit. These results distinguish the availability of visual information from its influence on prediction, and show that standard accuracy can coexist with limited sensitivity to the tested visual detail. They motivate training and evaluation that make answer-relevant visual distinctions necessary to answer correctly.
