CRISPR: Context-Refined Information Spatial Pooling with Region-awareness for Efficient Visual Token Compression in VLMs

Abstract

Vision-language models (VLMs) generate hundreds to thousands of visual tokens per image, leading to high computational cost and memory consumption during multimodal reasoning. A common efficiency strategy is to reduce input resolution; however, pixel-level downsampling often removes fine-grained semantic information and degrades performance on downstream tasks. We propose CRISPR, a context-refined information spatial pooling framework with region-awareness for visual token compression. Instead of reducing image resolution, CRISPR operates on encoded visual representations and compresses them into a smaller set of topology-preserving tokens. A Token Refiner first enriches local tokens with global context through self-attention, followed by a Local Token Compressor that performs block-wise cross-attention aggregation while maintaining spatial coherence, and a Global Token Fusion module that enables information exchange among compressed tokens. CRISPR supports controllable efficiency--accuracy trade-offs through the block-size parameter; in this paper, we evaluate 9× and 16× token reduction. Extensive experiments on multimodal benchmarks, including visual question answering and image captioning, demonstrate that CRISPR consistently outperforms low-resolution baselines at a comparable compressed sequence length. The proposed approach reduces both inference FLOPs and prefill latency.