SparseVLM+: Visual Token Sparsification With Improved Text-Visual Attention Pattern.
TL;DR
Improved text-visual attention patterns are introduced to enhance the fidelity of query-aware vision token selection and the Attention Gravity effect is correct, and a rank-based strategy to adaptively determine the sparsification ratio for each layer is introduced.
Abstract
In vision-language models (VLMs), visual tokens usually bear a significant amount of computational overhead despite sparsity of information in them when compared to text tokens. To address this, most existing methods learn a network to prune redundant visual tokens using certain training data. Differently, we propose a text-guided training-free token optimization mechanism dubbed SparseVLM without fine-tuning costs. Given that visual tokens complement text tokens in VLM's linguistic reasoning, we select relevant text tokens to rate the significance of visual tokens using self-attention matrices and, then, prune visual tokens using the proposed strategy to maximize sparsity while retaining information. In particular, we introduce a rank-based strategy to adaptively determine the sparsification ratio for each layer, alongside a token recycling method that compresses pruned tokens into more compact representations. However, the Attention Gravity and Attention Sink phenomena undermine the sparsification reference of SparseVLM: text-visual alignment. To address this, we introduce improved text-visual attention patterns to enhance the fidelity of query-aware vision token selection. First, we correct the Attention Gravity effect, where positional encodings induce disproportionately high attention to vision tokens positioned near textual tokens. We further mitigate the Attention Sink by selecting priority attention heads that suppress the excessive attention absorbed by semantically irrelevant vision tokens. The two parts refine the text-visual alignment from both the token and head perspectives. Finally, for video understanding tasks, we propose Temporal-Balanced Sparsification, complementing spatial relevance with temporal diversity to reduce cross-frame redundancy. Extensive experiments on 23 widely used image and video understanding tasks across 7 vision-language models demonstrate the effectiveness of our SparseVLM+ framework. For example, equipping LLaVA with SparseVLM+ yields a 57.1% reduction in FLOPs and a 38.4% decrease in CUDA latency, while preserving 99.6% of its original accuracy on average. Our code is available at https://github.com/Gumpest/SparseVLMs.