SG-MoE: Efficient Visual Compression for Multimodal Large Language Models via Semantic-Guided Mixture-of-Experts
Abstract
With the rapid advancement of Multimodal Large Language Models (MLLMs), the redundancy of visual tokens in images has become a critical bottleneck constraining inference efficiency. Existing resampling methods often suffer from detail loss and fail to dynamically adapt to user instructions. In this paper, we propose SG-MoE, a novel architecture designed for extreme vision token compression with robust cross-modal alignment. Specifically, we incorporate a semantic guidance mechanism that utilizes textual instructions to highlight relevant visual information. Subsequently, a heterogeneous Mixture-of-Experts (MoE) framework is employed to enrich multi-scale features and adaptively compress tokens based on image content. Furthermore, to compensate for potential loss during aggressive compression, we design a semantic-driven detail recovery strategy that restores fine-grained details risk being discarded. Extensive experiments demonstrate that our method achieves an aggressive compression ratio of ≈89% while maintaining 99.65% of the original performance across 9 benchmarks, even surpassing the uncompressed baseline on 5. This architecture offers an efficient and robust paradigm for constructing high-performance MLLMs.