Skip to content

One Model to Translate Them All? A Journey to Mount Doom for Multilingual Model Merging

Apr 2026 · arXiv.org · Vol abs/2604.02881 · 2 citations · 65 references
Computer Science

TL;DR

It is suggested that multilingual merging failures arise from target-side geometric misalignment within shared computational units, challenging the assumptions underlying standard weight-space merging for multilingual translation.

Abstract

Weight-space model merging combines independently fine-tuned checkpoints without access to the original training data. While merging has shown promise in multitask settings, its behavior in multilingual generative systems remains underexplored. We systematically study weight-space merging for multilingual machine translation by fully fine-tuning language models on large-scale bilingual corpora and evaluating representative merging strategies across shared-source, shared-target, and bidirectional consolidation settings. Our experiments reveal a strong directional asymmetry. Merging is comparatively more effective when models share a target language, improving multilingual coverage over the base model, but it still fails to preserve the peak performance of language-specific checkpoints. In contrast, when target languages differ, performance degrades sharply, especially in shared-source and bidirectional settings. To explain this behavior, we analyze internal representations and find that fine-tuning does not create disjoint language-specific sub-networks. Instead, independently fine-tuned models activate largely overlapping neurons while reshaping upper-layer target-generation representations into incompatible geometries. These findings suggest that multilingual merging failures arise from target-side geometric misalignment within shared computational units, challenging the assumptions underlying standard weight-space merging for multilingual translation. We make the code publicly available at https://github.com/babangain/mt-model-merging

View source

Similar papers

#artificial intelligence Preprint Sep 2026

CAMFT: Conflict-Aware Mergeable Fine-Tuning for Large Language Models

CAMFT is proposed, a Conflict-Aware Mergeable Fine-Tuning method that makes task adaptation both efficient and mergeaware and treats mergeability as a property shaped during fine-tuning, rather than only a problem to be solved after fine-tuning.

Jin-Gang Zhou, Hai-Yang Guo, Yuan-Pei Ma et al. · 1 citation
#natural language process... Open access Sep 2026

To Each Language Its Tokenizer: Modular Tokenizers for Efficient Multilingual LLMs

Multilingual Large Language Models (LLMs) traditionally rely on a single vocabulary shared by all supported languages, which can lead to uneven compression across them. Moreover, their large embedding and output matrices increase memory usage and slow inference, notably for small-scale models. It is also wasteful as mo...

Franck Signe Talla, Hippolyte Pilchen, François Yvon et al. · 0 citations

Sequential Adapter Stacking for Cross-Lingual Low-Resource ASR

This work investigates parameter-efficient approaches for transferring knowledge from resource-rich source languages to low-resource target languages on Whisper, and proposes Sequential Adapter Stacking, which places a trainable target-language adapter on top of a frozen source-language adapter.

Thai-Thinh Dang, Meng-Jie Qian, Kate Knill · 1 citation
#artificial intelligence Preprint Sep 2026

Efficient One-to-Many Translation with Joint Multi-Stream Diffusion

This work explores how diffusion can enable multilingual translation with a discrete diffusion framework that refines all target languages in parallel, achieving sublinear latency scaling with the number of targets, and supports deployment as a single unified model to replace multiple independent systems.

Yi-Wen Guan, Jacob Whitehill · 0 citations
#artificial intelligence Preprint Sep 2026

Beyond Cross-Lingual Transfer: Benchmarking Propagation Boundaries in Multilingual LLM Unlearning

CLLPU (Cross-Lingual and Language-Bound Protocol for LLM Unlearning), a multilingual benchmark that formulates this problem through two settings: common-goal forgetting, where target knowledge should be suppressed across all languages, and language-conditioned forgetting, where suppression should remain confined to a d...

Pengyang Shao, Chuan-Peng Lu, Wei Qin et al. · 2 citations

Data-Efficient Adaptation of Multilingual LLMs to Ukrainian

This work combines vocabulary surgery for tokenizer adaptation without full retraining, cross-lingual transfer of quality classifiers via translation, enabling filtering without target-language annotations, and generation of instruction data through translation, task conversion, and targeted synthesis.

Yurii Paniv, Bohdan Didenko, Mykola Haltiuk et al. · 4 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.