Skip to content
Preprint

Adaptive Two-Stage Visual Token Pruning for Efficient Inference in Video-Language Models

Aug 2026 · 1 citation · 32 references
Computer Science

TL;DR

This work proposes a two-stage adaptive token pruning strategy specifically designed for video processing that improves accuracy by +7\% on a video captioning benchmark at 10% token retention, while reducing computation TFLOPs by 95\%.

Abstract

Vision-language models excel at image and video understanding but suffer from high inference latency due to the need to process thousands of tokens per image, limiting their deployment on resource-constrained edge devices and in real-time surveillance applications. This challenge is further amplified in video processing, where multiple frames must be analyzed simultaneously. Existing token reduction techniques are largely developed for single-image inputs and therefore fail to account for the temporal and inter-frame redundancies present in video sequences. In addition, these methods generally rely on a fixed, uniform pruning ratio applied across all inputs, which is suboptimal because the degree of redundancy can vary significantly between different videos, necessitating content-dependent pruning levels to preserve critical information. To address these limitations, we propose a two-stage adaptive token pruning strategy specifically designed for video processing. In the first stage, we prune out the redundant frames, and in the second stage, token-level pruning is applied within the retained frames. Crucially, the pruning ratio in the second stage is determined adaptively based on the content of each video. This is achieved by analyzing the correlation structure of token embeddings to quantify redundancy, which is used to determine the ratio. Importantly, our method is entirely post-hoc and requires no additional training or fine-tuning, while achieving strong empirical gains; notably, it improves accuracy by +7\% on a video captioning benchmark at 10\% token retention, while reducing computation TFLOPs by 95\%.

View source

Similar papers

Preprint Aug 2026

Multi-Image Visual Token Pruning in Large Visual Language Models

A training-free, Adaptive Visual Token Pruning (AVTP) framework, applicable to diverse LVLM architectures is proposed, and adaptive pruning ratios in multi-image contexts where images of higher importance retain proportionally more tokens are implemented.

Rongyang Zhang, Cheng-Qiang Lu, Cong Li et al. · 0 citations
#computer vision Review Sep 2026

Why Is Video Still So Expensive? A Survey of Inference-Efficiency Mechanisms in Video and Audiovisual LLMs

This survey covers inference-efficiency mechanisms for visual and audiovisual VideoLLMs that report concrete reductions in parameter count, FLOPs per input, latency, memory, or visual and audio token count, and organize methods by the pipeline stage at which they act.

Killian Steunou, Yannis Tevissen, M. El Yacoubi · 0 citations
Conference Open access Sep 2026

CoFiE: Coarse-to-Fine Evidence Selection for Efficient Streaming Video Understanding

CoFiE is proposed, a Coarse-to-Fine Evidence Selection framework that decouples evidence selection into a coarse, query-agnostic filtering stage before the vision encoder and a fine, query-specific refinement stage during LLM prefill that outperforms strong open-source multimodal models while improving end-to-end infer...

Jing-Chi Jiang, Yi-Ran Ling, Ruo-Nan Li et al. · 1 citation

VideoMM: Adaptive Macro-Micro Inference for Efficient Video MLLMs

VideoMM is introduced, which marks a paradigm shift from model-centric downsizing to adaptive perceptual granularity and accelerates inference by 2.73 times over current leading methods, establishing a highly scalable paradigm for long-video understanding.

Hao-Yu Guo, Yuan Feng, Jun-Lin Lv et al. · 1 citation
#small language model Preprint Aug 2026

Training-Free Temporal Abstraction for General Video Understanding

STITCH is presented, a training-free method that divides a video into semantically meaningful temporal chunks that are computed once per video and reused across tasks, suggesting that reusable temporal abstraction is a promising direction for general video understanding.

Etienne Casanova, S. Brodjian, Pietro Perona · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.