Skip to content
Preprint

MoETok: Sparse Expert Routing for 1D Visual Tokenizers

Oct 2026 · 0 citations · 82 references
Computer Science

TL;DR

MoETok is introduced, a 1D visual tokenizer that routes tokens through sparsely activated experts that combines competitive visual understanding with state-of-the-art generation among unified discrete models, matching or surpassing diffusion-based models such as BAGEL and Qwen-Image on benchmarks.

Abstract

Modern visual tokenizers aim to preserve fine-grained visual details while learning high-level semantic representations for downstream generation and understanding. Query-based 1D visual tokenizers encode images into flexible latent sequences, yet how individual learnable queries gather and encode image information remains poorly understood. We reveal that these queries spontaneously develop complementary roles under semantic supervision: a small set of Content queries captures high-level semantic information, while a larger set of Detail queries preserves fine-grained visual details for reconstruction. To adapt token processing to these heterogeneous roles, we introduce MoETok, a 1D visual tokenizer that routes tokens through sparsely activated experts. Jointly optimized with reconstruction and semantic objectives, the router learns role-specialized processing paths adapted to each token's representation. Across model scales and token budgets, MoETok consistently improves reconstruction, semantic representation, and autoregressive generation over dense tokenizers under matched supervision and active computation. On ImageNet, it combines strong reconstruction and semantic representation with state-of-the-art class-conditional generation within the 1D tokenizer family. We further validate MoETok at scale in a unified autoregressive model jointly trained for understanding and generation with a next-token cross-entropy objective. The resulting model combines competitive visual understanding with state-of-the-art generation among unified discrete models, matching or surpassing diffusion-based models such as BAGEL and Qwen-Image on benchmarks including DPG-Bench (87.61), GenEval2 (40.04 GM), and WISE (0.49).

View source

Similar papers

#artificial intelligence Preprint Oct 2026

VisionWeave: Weaving Elastic Visual Representations as a Native Capability of MLLMs

Multimodal large language models have become the dominant paradigm for visual understanding, but incur substantial costs by encoding inputs into dense, fixed-size patch tokens. However, visual information is unevenly distributed: some regions require fine-grained detail, while others admit compact representations. Down...

Yuan Feng, Qi-Ze Yang, Rui-Zhe Chen et al. · 1 citation
Preprint Nov 2026

Semantic-Guided Geometric Decoding for Token-Efficient Visual Grounding

Multimodal large language models (MLLMs) provide a unified interface for language-conditioned visual perception. In pixel-to-sequence grounding, autoregressive coordinate generation requires multiple sequential decoding steps per box, increasing inference cost with the number of instances. Pixel-to-embedding methods re...

Zhao-Jie Chen, Jun-Yan Yang, Lei-Chao Yang et al. · 0 citations
Book Open access Nov 2026

QTok: Scalable 1D Visual Tokenization via Decoupled Querying

One-dimensional (1D) image tokenizers have emerged as key enablers for efficient visual generation by compressing images into compact discrete sequences. However, existing approaches adopt a coupled architecture that concatenates learnable latent tokens with image patches at the input. This tight entanglement creates t...

Wen-Zhao Xiang, Hai-Feng Ma, Yueh-Tse Wu et al. · 0 citations
#artificial intelligence Book Open access Oct 2026

LearnPrunerV2: Learnable Progressive Token Pruning for Vision-Language Models

Vision-Language Models (VLMs) have demonstrated remarkable multimodal capabilities but suffer from prohibitive computational overhead due to the massive number of visual tokens. While token pruning is a promising solution, existing methods still suffer from two major drawbacks: 1) unreliable pruning criteria that fail...

Rinyoichi Takezoe, Ya-Qian Li, Wei-Ren Wang et al. · 0 citations
#artificial intelligence Preprint Nov 2026

BtsPruner: Role-Aware Visual Token Pruning for Large Vision-Language Models

Large Vision-Language Models (LVLMs) often process hundreds or thousands of visual tokens, resulting in substantial computational and memory overhead during inference. Existing training-free pruning methods typically estimate token importance using language-model attention, visual attention, or feature similarity. Howe...

Zi-Hao Qi, Zhi-Bo Zhang, Miao-Xin Zhang et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Mixture of Layers: Dynamic Layer Routing for Visual Reasoning

Pre-trained vision encoders contain layer-wise visual representations that differ in spatial granularity, semantic abstraction, and sensitivity to local details. However, most Multimodal Large Language Models (MLLMs) rely on only the final or penultimate vision encoder representations or fixed aggregation rules, making...

Jeonghwan Kim, Sofia Stoica, Ji-Wan Chung et al. · 2 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.