Multimodal large language models have advanced rapidly, yet most remain English-centric, as scaling multilingual multimodal instruction tuning is limited by the scarcity and high cost of high-quality non-English image-text supervision. Although multilingual text data is abundant, naive textual fine-tuning can disrupt vision-language alignment and induce catastrophic forgetting. We propose Vision-Free Adaptation (VFA), a framework that decouples multilingual language enhancement from visual alignment by composing complementary task vectors over a shared LLM backbone. Specifically, we fine-tune a base LLM on multilingual text data to derive a multilingual task vector, which is then merged with the vision-aligned task vector of an MLLM. Experiments on five MLLMs across six multilingual multimodal benchmarks show consistent improvements while preserving both general multimodal and text-only capabilities. Moreover, using less than 2% of the text data, VFA narrows the gap to the fully multimodal-trained model, demonstrating its data efficiency.
Yixia Li, Yaqing Shi, Zhiwen Ruan et al.· Annual Meeting of the Associ...· 0 citations
Adapting instruction-tuned large language models (LLMs) to downstream domains is increasingly common, yet fine-tuning on imperfect data can erode the safety alignment learned during post-training. Existing safety-preserving fine-tuning methods typically optimize the aligned instruction model directly, which can destabilize refusal behaviors or impose an ''alignment tax'' that limits task adaptation. We propose SAFT (Safety-preserving Adaptation via Fine-tuning Transfer), a safety-preserving adaptation framework that decouples task learning from alignment preservation by learning a safety-guided task update on the paired pretrained base model, rectifying task gradients to avoid conflicting directions with respect to a safety objective, and then transferring the update to the frozen instruction model via parameter-space grafting. Across mathematical reasoning, code generation, and medical question answering on two open-source model families (Llama3.1-8B-Instruct and Gemma3-4B-IT), SAFT improves downstream utility while maintaining low harmfulness under a unified evaluation protocol, and achieves better safety and utility trade-offs than nine baselines. Warning: This paper contains unfiltered content generated by LLMs that may be offensive to readers.
Zhiwen Ruan, Yan Yang, Zhuocheng Liang et al.· Proceedings of the 32nd ACM...· 0 citations