Empirical analyses show that MoSE improves high-, medium-, and low-resource languages simultaneously, with the largest gains on low-resource speech, thereby breaking the curse of multilinguality without compromising high-resource performance.
Abstract
Multimodal large language models (MLLMs) have achieved significant success in speech-to-text translation (S2TT). However, when processing multilingual speech inputs, a single speech encoder shared across all languages suffers from the curse of multilinguality: languages at different resource levels compete for limited representation capacity, leading to strong high-resource performance but substantial degradation on low-resource speech. To address this problem and improve multilingual consistency, we propose MSRT, a novel framework built around a resource-aware Mixture of Speech Encoders (MoSE). MoSE uses an explicit language router to assign each utterance to an appropriate expert encoder. A frozen expert preserves high-resource language capabilities, while a trainable expert adapts to and specializes in medium- and low-resource languages. We further introduce a five-stage curriculum learning strategy that substantially reduces data dependence, requiring only 10 hours of paired S2TT data per language for effective alignment. We conduct extensive experiments on 45 languages, systematically evaluating all $45 \times 44$ translation directions. Our 4B-parameter model achieves state-of-the-art performance, outperforming substantially larger baselines. Empirical analyses show that MoSE improves high-, medium-, and low-resource languages simultaneously, with the largest gains on low-resource speech, thereby breaking the curse of multilinguality without compromising high-resource performance. To support future multilingual S2TT research, we release our code and models.
In every setting, pre-adaptation on related auxiliary languages yields no practically meaningful improvements once as little as one hour of target-language data is available, suggesting that relatedness alone may not reliably predict transfer gains in large multilingual ASR, or constitute an effective strategy for extending such models to low-resource languages.
A. Florian, C. Amol, Hope Kerubo Ombaba et al.· 0 citations
Neural transducers offer an alignment-free framework for speech-to-text modeling, and hierarchical transducer architectures further improve multilingual joint automatic speech recognition (ASR) and speech translation (ST) by stacking a translation-focused encoder on top of an ASR encoder. However, extending hierarchical transducers to multilingual many-to-many settings remains challenging: fully shared models often suffer from negative transfer and unstable target-language generation, while training separate models for each direction is computationally prohibitive. We propose LCMA-SRT (Language-Conditional Mixture-of-Experts Adapters for Speech Recognition and Translation), which augments a hierarchical transducer with language-conditional Mixture-of-Experts (MoE) adapters. A source-conditioned MoE adapter (SRC-MoE) uses source-language embeddings to reduce cross-language interference and improve multilingual ASR. A target-conditioned MoE adapter (TGT-MoE) uses the desired target language to reduce cross-target interference and stabilize target-language generation in many-to-many ST. Experiments on Europarl-ST (9 languages, 72 directions) show that LCMA-SRT improves both ASR and ST within a single joint model, reducing average WER and improving BLEU and COMET over strong hierarchical transducer baselines. We release our code and models at https:
Nan Li, Xiaoyong Guo, Hao Huang et al.· Annual Meeting of the Associ...· 0 citations
Speech-to-Speech Translation (S2ST) is a vital technology for facilitating communication across languages, especially in multilingual and resource-limited environments. However, it is still hard to make reliable S2ST systems for low-resource languages like Telugu because there aren’t many parallel speech resources, the morphology is complex, the speakers are different, and the real world is noisy. This paper introduces a Telugu-adapted cascaded S2ST framework that incorporates IndicWhisper for automatic speech recognition, IndicTrans2 for machine translation, and text-to-speech synthesis for generating spoken output. The primary contribution of this work is the integration of Telugu-aware preprocessing and post-processing techniques, encompassing audio normalization, noise reduction, sentence-level segmentation, Telugu script normalization, numerical normalization, and named-entity preservation. These language-aware parts are meant to make transcription more reliable and translation more accurate when there aren’t many low resources.We test the proposed system with benchmark and custom datasets, such as TeluguST-46, the BhashaAnuvaad subset, and recordings from the real world that are noisy. The full system gets a BLEU score of 0.62, a WER of 0.21, and a chrF score of 0.89, which is better than a plain cascade baseline. Ablation analysis further substantiates that both preprocessing and post-processing significantly influence the final performance. The research illustrates that language-aware adaptation enhances cascaded S2ST systems for Telugu and other low-resource languages.
Navya Guggilam, Vijaya Babu Panthagani, Bollimunta Ankammarao et al.· 2026 7th International Confe...· 0 citations
This work formalizes joint speech summarization and translation (JSumT), the generation of a succinct, faithful target-language summary directly from a long spoken document in a source language, and establishes a foundation for developing and evaluating multilingual systems capable of jointly interpreting, compressing, and translating long-form speech.
Yejin Jeon, Marie Maltais, Virginia Ceccatelli et al.· 0 citations
Recent advances in zero-shot text-to-speech (TTS) have substantially improved speech quality and voice cloning fidelity. However, many zero-shot TTS systems still depend on audio prompt transcripts at inference time. This dependency limits cross-lingual voice cloning, since in-the-wild reference audio is often untranscribed. In this technical report, we present Confucius4-TTS, a multilingual zero-shot TTS system that supports 14 languages and performs both intra-lingual and cross-lingual reference cloning without requiring transcripts of audio prompts. Confucius4-TTS follows a two-stage architecture, consisting of text-to-semantic (T2S) and semantic-to-acoustic (S2A) modules. The LLM-based T2S module uses a learnable speaker encoder to extract timbre features from self-supervised speech representations, and the conditional flow-matching S2A module converts the predicted semantic tokens into mel-spectrograms. The same model also supports continuation cloning when a reference transcript is available. Confucius4-TTS is trained on large-scale multilingual speech data. It achieves high intelligibility and speaker similarity on public benchmarks. On the CV3-Eval cross-lingual benchmark, Confucius4-TTS obtains an average WER of 3.73% across six directions. On our internal cross-lingual set, it achieves the best average overall rank in human evaluation among recent open-source and commercial systems. We release code, model checkpoints, and demos at https://github.com/netease-youdao/Confucius4-TTS.
Huaxuan Wang, Huimin Wang, Ruiyu Zhang et al.· 0 citations