JoinTok: A Language-Model-Joint Unified Tokenizer for Understanding and Reconstruction
Abstract
A visual tokenizer supporting both understanding and reconstruction is the fundamental cornerstone for Unified Multimodal Models (UMMs). Recent works typically decouple the visual tokenizer from Vision-Language Models (VLMs) to optimize it for both tasks. However, the resulting tokenizer fails to support plug-and-play reuse of the VLM's understanding capabilities, demanding costly LLM retraining. This occurs because the decoupled training inevitably disrupts the inherent alignment between the tokenizer and the LLM. To overcome this limitation, we propose JoinTok, a novel LLM-training-free paradigm that jointly optimizes LLM's understanding and pixel-level reconstruction through a dual-projection mechanism. Instead of compromising existing capabilities, JoinTok explicitly preserves the pre-trained VLM's comprehension capacity as an optimization objective while injecting reconstruction skills, ultimately leading to enhanced understanding proficiency. Crucially, we uncover the inherent orthogonality between reconstruction and understanding, providing a theoretical basis for their joint optimization without mutual interference. Extensive experiments show that JoinTok consistently surpasses state-of-the-art methods on comprehensive visual understanding and reconstruction benchmarks.