TCSP: Text-Conditioned Spectral Prototyping for Multimodal Token Compression

Abstract

The inference performance of Multimodal Large Language Models (MLLMs) typically relies on a massive number of visual block tokens, leading to substantial computational overhead. However, existing training-free compression methods often sacrifice semantic relevance, disrupt spatial coherence, or require extensive architectural modifications. Consequently, we propose Text-Conditioned Spectral Prototyping for Multimodal Token Compression (TCSP). We construct a joint affinity matrix based on both text relevance and visual similarity to derive boundary-continuous and semantically aligned segments via spectral embedding, subsequently replacing redundant information within these segments with weighted prototypes to reduce visual tokens for downstream inference. This method is entirely training-free and plug-and-play, allowing it to be inserted into intermediate layers of the vision encoder or before/after the projector module as needed. Evaluations on LLaVA-1.5-7B demonstrate that after a approximately $3 \times$ reduction in visual tokens, the overall performance remains baseline levels, even achieving marginal denoising-like improvements in several general QA benchmarks. More aggressive compression results in a graceful performance degradation, whereas tasks emphasizing compositional reasoning exhibit higher sensitivity to the compression scale. TCSP integrates text-awareness, spectral partitioning, and prototype replacement within a minimally invasive design. It achieves high, near-lossless compression ratios and a robust accuracy-efficiency trade-off without training, providing a concise and reliable pathway for general reasoning and resource-constrained deployment.