The results show that LVLM predictions are sensitive to lexical, OCR-derived and stylistic cues, with injected surface signals causing substantial changes in model predictions despite unchanged underlying image-text relationships.
Abstract
Large Vision-Language Models (LVLMs) have demonstrated strong performance on multimodal benchmarks, yet it remains unclear whether they genuinely reason about relationships between images and text or rely on superficial correlations, known as shortcut learning. This question is particularly important for multimodal sarcasm detection, where successful prediction depends on recognizing pragmatic incongruity rather than treating sarcasm as simple image-text mismatch. We introduce PragMatch, a controlled benchmark of 3,000 image-text pairs derived from MMSD2.0, including original sarcastic examples and constructed literal and hard-negative pairs. We identify influential shortcut cues through systematic masking and evaluate their impact through targeted injection experiments. Our results show that LVLM predictions are sensitive to lexical, OCR-derived and stylistic cues, with injected surface signals causing substantial changes in model predictions despite unchanged underlying image-text relationships. Our findings reveal limitations in current LVLMs while PragMatch provides a systematic testbed for evaluating multimodal pragmatic reasoning beyond surface-level image-text alignment.
This work investigates how decoder-only transformers resolve lexical ambiguity through layer-by-layer analysis of three models spanning three parameter sizes, finding that representations become maximally distinct in middle layers, then partially reconverge in late layers, while the KL divergence between their next-token predictions reaches its maximum in the final layers.
K. Scott, Narun Pat, Veronica Liesaputra· 0 citations
Vision-language models (VLMs) have achieved strong performance on tasks such as image captioning, visual question answering, and image-to-text generation. However, they are predominantly trained on English-centric data, which limits their ability to handle culturally grounded visual understanding and leads to failures in interpreting region-specific meanings, symbolic content, and context-dependent visual cues. Existing benchmarks for cultural competence are often template-driven and focused on surface-level recognition, making them insufficient for evaluating deeper linguistic and pragmatic understanding in culturally situated settings. We introduce PoVisLE, a monocultural vision-language benchmark for Polish designed to evaluate culturally grounded multimodal understanding under a grounded evaluation paradigm, where language is interpreted in interaction with visual context. The dataset contains 1,117 images and 2,366 manually annotated VQA pairs. Overall, our dataset provides a controlled and challenging resource for assessing culturally grounded vision-language understanding beyond surface-level recognition.
Anna Kołos, Grzegorz Statkiewicz, Karolina Seweryn et al.· 1 citation
LookBack, a training-free LVLM response scoring method that augments token likelihood with visual lookback score, a lightweight measure of how strongly each response token refers to image tokens, consistently improves Best-of-$N$ selection over existing baselines with negligible additional overhead.
Beomsik Cho, Jinhyeong Kim, Dongseok Lee et al.· 0 citations
Vision-language models can exhibit visual concept-conditioned divergence: given images containing demographic features, corporate logos, or ideological symbols, some models produce unusually uniform responses that differ from what peer models say about the same input. These behaviors evade text-only audits because visual concepts cannot be isolated or substituted the way text tokens can. We present VISTA (Visual Inconsistency Screening Through Analysis), a black-box cross-model audit that couples semantic entropy with distribution-based divergence to flag model-specific anomalies. In a controlled study, we implant concept-conditioned stances in three VLMs via fine-tuning on small biased datasets and confirm that VISTA detects them. Auditing six VLMs across 19 topics, VISTA surfaces 142 high-suspicion cases (1.2%) and identifies selective refusal as a previously unreported divergence pattern, where models refuse demographic queries at rates varying from 0 to 65% across groups.
Multimodal large language models (MLLMs) exhibit substantial performance degradation in non-English visual reasoning, despite the strong multilingual competence of their text-only backbones. While mechanistic evidence from text-only models suggests that non-English inputs are routed through an English-centric latent space, the multimodal implications of this phenomenon remain unexplored. Through rigorous mechanistic analysis, we identify the \textbf{Ghost Anchor} phenomenon: a temporal modality asynchrony where linguistic translation to the English semantic manifold completes in early layers, while visual semanticization remains immature. Consequently, visual signals are physically present yet functionally invisible during the early alignment window. To rectify this, we propose \textbf{ANCHOR}, a training framework employing Proactive Visual Anchoring (PVA) to accelerate early visual semantic emergence, ensuring visual representations proactively guide linguistic translation. Mechanistic interventions confirm that ANCHOR successfully restores the causal influence of visual signals during early translation. Furthermore, extensive experiments on XMMMU, MaXM, and CVQA demonstrate that ANCHOR consistently outperforms standard baselines, achieving robust visual reasoning across both fine-tuned and zero-shot languages.
Yihang Du, Juhao Liang, Zheng-Zhao Lai et al.· 0 citations
Multimodal large language models (MLLMs) have achieved strong performance in general visual question answering, yet their visual faithfulness in handling discrete symbolic information remains insufficiently understood. Symbols such as numbers, time expressions, identifiers, license plates, and alphanumeric strings impose low semantic redundancy and strict character-level constraints, making overall VQA accuracy or isolated OCR-style evaluation inadequate for diagnosing model reliability. To address this gap, this paper proposes a cross-regime diagnostic framework for evaluating symbol consistency in MLLMs. Under a unified protocol, we evaluate five representative models, including BLIP-2, InstructBLIP, LLaVA, InternVL, and Qwen, across VQA, TextVQA, a self-constructed Symbol Subset, Regime 2a with uncontrolled generative symbol rendering, and Regime 2b with controlled clear-symbol grounding. We further introduce a 2a $\rightarrow 2$ b paired recovery analysis to distinguish rendering-sensitive errors caused by upstream symbol degradation from persistent errors that remain under clear visual evidence. Regime 2a is interpreted as an uncontrolled stress probe rather than a clean OCR benchmark, and its accuracy reflects both upstream rendering quality and downstream model reading. Experimental results show that general VQA accuracy can substantially overestimate symbol-centered reliability, especially for weaker models. Symbol consistency failures are not merely OCR recognition errors, but arise from the interaction of target-region binding, character-faithful transcription, answer completeness, and language-prior normalization. Although clear-symbol conditions improve stronger models, persistent failures remain in character-level grounding, target binding, and task following. This study separates symbol consistency from general VQA evaluation and reveals its multi-stage failure mechanisms, offering a reusable diagnostic perspective for reliability assessment and symbol-capability improvement in MLLMs.