Unified Multimodal Model Proficient in Continued Training
Abstract
Unified large multimodal models, capable of both understanding and generation, face significant challenges during continued training without access to original pre-training data. This post-pre-training task, critical for enhancing core foundational capabilities, is plagued by catastrophic forgetting and compounded by task interference inherent in unified architectures. To address this, we propose MEANING, a novel model that unifies sparse adaptation with targeted knowledge distillation. We introduce a Dual Layer-Level LoRA Routing mechanism, where modality-specific routers dynamically activate a sparse, input-dependent subset of LoRA adapters to mitigate task interference. Crucially, this sparse adaptation is coupled with a novel Layer-Routed Attention Distillation loss, which aligns internal reasoning patterns exclusively within these active layers. This integrated approach simultaneously teaches the model where to adapt and what to learn. Experiments show MEANING prevents the catastrophic forgetting seen in naive continued training (\eg, an average 60.1% performance drop across understanding benchmarks) and successfully enhances both faculties, which has been verified across multiple tasks spanning understanding, generation, and editing.