This work proposes a training-free, hierarchical token pruning framework that utilizes query guidance prior to executing any LLM blocks, and demonstrates that this plug-and-play framework retains 98.0% and 94.8% of unpruned performance at 25% and 10% vision token retention, respectively.
Abstract
High-resolution Large Vision-Language Models (LVLMs) preserve critical, fine-grained visual details by encoding a global thumbnail alongside multiple local crops. However, the resulting thousands of vision tokens introduce severe computational bottlenecks during Large Language Model (LLM) inference. Existing token pruning methods rely primarily on query-agnostic visual saliency, while current query-aware approaches either operate at a coarse crop-level resolution or compute token relevance too late, after vision tokens have already entered the heavy LLM layers. To address this, we propose a training-free, hierarchical token pruning framework that utilizes query guidance prior to executing any LLM blocks. Our top-down hierarchy first operates at the instance-level, dynamically scaling the vision token retention ratio of each sample based on its visual complexity. Next, at the crop-level, we distribute this sample budget across local crops using sentence-level cross-modal similarity. Finally, at the token-level, we exploit projected vision tokens alongside textual query token embeddings in the shared space to measure fine-grained alignment, applying a text-debiasing normalization to produce high-fidelity query scores. These query-guided budgets and scores are fused with local and global visual saliency metrics to select the most critical vision tokens. Extensive experiments on LLaVA-NeXT-7B demonstrate that our plug-and-play framework retains 98.0% and 94.8% of unpruned performance at 25% and 10% vision token retention, respectively. Notably, at 25% retention, our method delivers a 2.0x inference throughput speedup and even surpasses the unpruned baseline on POPE by 0.7 points.
The massive redundancy of visual tokens in Vision-Language Models (VLMs) creates a critical computational bottleneck during large language model (LLM) prefill. Recent training-free pruning methods provide plug-and-play acceleration, yet their pruning decisions often rely on limited visual states or delayed in-LLM signa...
Qing-Xin Liu, Jin-Zhi Liao, Xiang Zhao· Proceedings of the 35th ACM...· 0 citations
TReVS is proposed, a training-free framework that combines textual relevance with vision-encoder saliency for pre-LLM pruning and leverages high-variance attention heads to remove task-irrelevant tokens at shallow-to-intermediate layers of the LLM.
Large Vision-Language Models (LVLMs) often process hundreds or thousands of visual tokens, resulting in substantial computational and memory overhead during inference. Existing training-free pruning methods typically estimate token importance using language-model attention, visual attention, or feature similarity. Howe...
Zi-Hao Qi, Zhi-Bo Zhang, Miao-Xin Zhang et al.· 0 citations
Large vision-language models (LVLMs) represent images and videos as long visual-token sequences, substantially increasing inference costs. These costs motivate visual-token pruning, whose effectiveness depends on both what signal is used for selection and when that signal becomes available. Our analysis reveals complem...
Intaek Shin, Geuntaek Lim, Sungjune Park et al.· 0 citations
High-resolution visual inputs can substantially enhance the fine-grained understanding of documents, charts, and complex layouts in multimodal large language models (MLLMs), but they also incur higher inference latency and greater memory overhead. Existing visual token compression methods primarily rely on pruning or m...
Hong-Fei Ren, Hu Zhang, Ying-Long Wang et al.· Proceedings of the 34th ACM...· 0 citations
Large vision-language models (VLMs), which process both visual inputs and text queries, incur high inference costs due to the large number of visual tokens, making visual token pruning a natural approach for improving efficiency. Recent work has leveraged query information to guide token selection, often through heuris...
Harshithanjani Athi, Sravan Kumar Ankireddy, J. Zhang et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.