It is suggested that multilingual merging failures arise from target-side geometric misalignment within shared computational units, challenging the assumptions underlying standard weight-space merging for multilingual translation.
Abstract
Weight-space model merging combines independently fine-tuned checkpoints without access to the original training data. While merging has shown promise in multitask settings, its behavior in multilingual generative systems remains underexplored. We systematically study weight-space merging for multilingual machine translation by fully fine-tuning language models on large-scale bilingual corpora and evaluating representative merging strategies across shared-source, shared-target, and bidirectional consolidation settings. Our experiments reveal a strong directional asymmetry. Merging is comparatively more effective when models share a target language, improving multilingual coverage over the base model, but it still fails to preserve the peak performance of language-specific checkpoints. In contrast, when target languages differ, performance degrades sharply, especially in shared-source and bidirectional settings. To explain this behavior, we analyze internal representations and find that fine-tuning does not create disjoint language-specific sub-networks. Instead, independently fine-tuned models activate largely overlapping neurons while reshaping upper-layer target-generation representations into incompatible geometries. These findings suggest that multilingual merging failures arise from target-side geometric misalignment within shared computational units, challenging the assumptions underlying standard weight-space merging for multilingual translation. We make the code publicly available at https://github.com/babangain/mt-model-merging
CAMFT is proposed, a Conflict-Aware Mergeable Fine-Tuning method that makes task adaptation both efficient and mergeaware and treats mergeability as a property shaped during fine-tuning, rather than only a problem to be solved after fine-tuning.
Jin-Gang Zhou, Hai-Yang Guo, Yuan-Pei Ma et al.· 1 citation
Multilingual Large Language Models (LLMs) traditionally rely on a single vocabulary shared by all supported languages, which can lead to uneven compression across them. Moreover, their large embedding and output matrices increase memory usage and slow inference, notably for small-scale models. It is also wasteful as mo...
Franck Signe Talla, Hippolyte Pilchen, François Yvon et al.· Conference on Empirical Meth...· 0 citations
This work investigates parameter-efficient approaches for transferring knowledge from resource-rich source languages to low-resource target languages on Whisper, and proposes Sequential Adapter Stacking, which places a trainable target-language adapter on top of a frozen source-language adapter.
Thai-Thinh Dang, Meng-Jie Qian, Kate Knill· arXiv.org· 1 citation
This work explores how diffusion can enable multilingual translation with a discrete diffusion framework that refines all target languages in parallel, achieving sublinear latency scaling with the number of targets, and supports deployment as a single unified model to replace multiple independent systems.
CLLPU (Cross-Lingual and Language-Bound Protocol for LLM Unlearning), a multilingual benchmark that formulates this problem through two settings: common-goal forgetting, where target knowledge should be suppressed across all languages, and language-conditioned forgetting, where suppression should remain confined to a d...
Pengyang Shao, Chuan-Peng Lu, Wei Qin et al.· 2 citations
This work combines vocabulary surgery for tokenizer adaptation without full retraining, cross-lingual transfer of quality classifiers via translation, enabling filtering without target-language annotations, and generation of instruction data through translation, task conversion, and targeted synthesis.
Yurii Paniv, Bohdan Didenko, Mykola Haltiuk et al.· 4 citations