MMSep: Efficient Multimodal Long-Generation Reasoning via Multimodal Separator Compression

Abstract

Existing research on Efficient Multimodal Large Language Models (EMLLMs) primarily focuses on reducing the number of visual tokens in the prefilling stage, which is tailored to short-answer inference scenario. However, in more complex multimodal reasoning tasks, models are often required to generate lengthy intermediate reasoning rationales, while repeatedly revisiting prefilled contexts to verify and revise reasoning paths. As the generation length increases, the cumulative overhead of the decoding stage rises surpasses that of pruned prefilling, to become the dominant cost source for end-to-end inference. Investigating decoding-time attention behaviors, we observe two phenomena on textual and visual side related to selective and effective memory retention. Based on these observations, we propose MMSep, a training-free multimodal separator localization and compression framework that improves efficiency in both prefilling and decoding. MMSep (i) localizes visual anchors/separators during prefilling via question-guided attention and a spatial–similarity constraint, and (ii) performs structured KV-cache compression during decoding by retaining textual separators as long-range context and enabling separator-triggered, on-demand visual recall. Experiments on four MLLM backbones across long-generation and standard reasoning benchmarks demonstrate that MMSep consistently reduces latency while maintaining competitive generation quality and reasoning accuracy. Our code is available at https://github.com/MeinhardMark/MMSep.