MERIT: Dynamics-Aware Visual Token Compression for High-Resolution Document Understanding in Multimodal Large Language Models
Abstract
High-resolution visual inputs can substantially enhance the fine-grained understanding of documents, charts, and complex layouts in multimodal large language models (MLLMs), but they also incur higher inference latency and greater memory overhead. Existing visual token compression methods primarily rely on pruning or merging strategies, yet they remain inadequate for document-centric scenarios: the former may erroneously discard low-saliency but semantically critical regions, such as small-font text and symbols, while the latter may attenuate the high-frequency local details essential for OCR through feature averaging.To address these limitations, we propose MERIT, a training-free visual token compression framework that estimates token importance through multi-view dynamics from global semantics, local structure, and signal saliency. Building on this design, we further develop Dynamics-Aware Proxy Compression (DAPC), which preserves high-value tokens via feature-faithful hard sampling and compensates for the influence of compressed regions in subsequent attention through quality redistribution. Experimental results demonstrate that MERIT achieves strong performance across document understanding, OCR, and general multimodal benchmarks. On LLaVA-OneVision-7B, MERIT retains 97.6% and 93.0% of the original performance at compression ratios of 66.7% and 77.8%, respectively, while reducing time-to-first-token (TTFT) by 62.9% and 60.7%, respectively, relative to the best competing baselines under the respective settings. These results suggest that dynamics-aware, feature-faithful compression provides an effective pathway toward efficient high-resolution multimodal document understanding.