Skip to content

Layers, Sinks, and Scaling: Adaptive Evidence Selection for Multimodal Large Language Models

Sep 2026 · 0 citations · 22 references
Computer Science

TL;DR

Adaptive Relevance-guided Evidence Allocation (AREA) is introduced, a training-free inference-time method that formulates evidence highlighting as adaptive allocation and establishes the best performance among training-free highlighting methods.

Abstract

Multimodal large language models (MLLMs) can answer knowledge-intensive visual questions by combining visual evidence from images with facts retrieved from external sources. However, MLLMs may overlook relevant evidence in both modalities, attending weakly to the textual sentences or visual regions needed for the correct answer. Recent efforts address this by highlighting retrieved text and marking visual regions before generation, but apply a fixed, one-shot policy that cannot adapt to three sources of variation: whether highlighting is necessary, how much evidence different examples require, and when different textual evidence becomes relevant as the answer unfolds. We introduce Adaptive Relevance-guided Evidence Allocation (AREA), a training-free inference-time method that formulates evidence highlighting as adaptive allocation. AREA generates a single probe token to read visual and textual relevance from fixed backbone layers, then makes three decisions: i) whether to intervene (controlled by natural attention coverage and visual sink contamination), ii) how much evidence to expose (determined by relevance entropy), and iii) when to refresh text during generation (triggered by causal context-attention peaks). Across four KB-VQA and seven standard multimodal benchmarks with nine frozen MLLM checkpoints, establishes the best performance among training-free highlighting methods.

View source

Similar papers

#natural language process... Open access Oct 2026

CLIMB: Confidence-Guided Complementary Evidence for Multimodal Retrieval-Augmented Generation

Multimodal large language models (MLLMs) have shown strong visual reasoning abilities, but knowledge-intensive visual question answering often requires external textual evidence beyond the image and the model's parametric knowledge. Existing multimodal RAG systems commonly rely on Top-$K$ retrieval or reranking, which...

Hang Gao, Wu-Jiang Xu, Zhixing Zhang et al. · 0 citations
Preprint Sep 2026

Pay More Attention To Text In High-Resolution MLLMs

Failures of high-resolution MLLMs are commonly attributed to a visual problem, motivating zooming, cropping, and related visual interventions to recover fine-grained evidence or suppress interference. Yet recent studies suggest that relevant visual evidence is already encoded in intermediate representations, indicating...

Z. Mao, Wen-Zhuo Zhao, Xian-Jie Liu et al. · 1 citation
Preprint Aug 2026

Question-Guided Evidence Acquisition for Multimodal Visual Question Answering

Q-Guide is built, a small agent that reads a question, works out what evidence it is still missing, and calls targeted tool(s) to recover it---reading text where text is needed, zooming in where detail is needed, or grounding a region where position matters.

A. Popa · 0 citations

Skip the Talk, Re-Focus on Vision: Latent Reasoning for Reasoning Segmentation in Multimodal Large Language Models

LIRSeg is proposed, which fully replaces explicit CoT with a compact set of learnable latent tokens for reasoning segmentation, and introduces three complementary mechanisms from an information perspective: extreme-advantage sampling for selecting informative training signals, decoupled exploration-stability updates fo...

Tian-Hang Guo, Yu-Lin He, Wei Chen et al. · 0 citations
#computer vision Sep 2026

MCD: Causal Distillation of Multimodal In-Context Learning in Large Vision-Language Models

Large vision-language models (LVLMs) exhibit strong multimodal in-context learning (ICL) capabilities, yet this ability degrades substantially as model size decreases. Knowledge distillation offers a natural way to bridge this gap, but existing methods primarily align output distributions or hidden representations dire...

Yanshu Li, Jia-Qian Li, Can-Ran Xiao et al. · 0 citations
#artificial intelligence Preprint Sep 2026

SinkPruner: Sink-Free Visual Token Pruning for Multimodal Large Language Models

SinkPruner is proposed, a training-free visual token pruning framework for efficient MLLM inference that follows a coarse-to-fine design with two key modules: a visual sanitizer that filters high-norm redundancies and alleviates attention sink and attention dispersion, and a text-guided pruner that further retains toke...

Shi-Yu Li, Zi-Yuan Hu, Shijia Huang et al. · 2 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.