This work proposes CAROT (Cross-Lingual Alignment of Representations in a Language-Agnostic Space via Optimal Transport), which consists of identifying language-specific representations in LLMs'internal states and aligning language-agnostic representations across languages at the token level by optimal transport, while explicitly preserving language-specific representations.
Abstract
Cross-lingual alignment (CLA) aims to align the representations of large language models (LLMs) across languages, enabling cross-lingual transfer to improve multilingual capabilities. Previous CLA methods often ignore language-specific information encoded in representations and only consider sentence-level alignment, which may lead to suboptimal performance and input-output language mismatch. We propose CAROT (Cross-Lingual Alignment of Representations in a Language-Agnostic Space via Optimal Transport), which consists of two steps: identifying language-specific representations in LLMs'internal states and aligning language-agnostic representations across languages at the token level by optimal transport, while explicitly preserving language-specific representations. Inference-time steering experiments show that the representations computed by CAROT are effective alignment targets, improving multilingual performance by up to 11.2 points in accuracy while maintaining input-output language consistency. We further use the representations obtained by CAROT as training targets, internalizing the aligned representations. The trained models outperform existing CLA methods in 11 of 18 evaluation settings (3 models $\times$ 3 tasks $\times$ ID/OOD languages). Our work provides insights into what constitutes effective alignment targets for CLA in LLMs. Code is available at https://github.com/ynklab/CAROT
This work proposes an approach that integrates code-switching directly into masked language model pretraining, and introduces a multiview probabilistic translation strategy that samples candidate translations based on alignment likelihoods, applying substitutions only to unmasked tokens.
Ruan Visser, Trienko L. Grobler, Marcel Dunaiski· 0 citations
It is confirmed that cross-lingual alignment can emerge from the structure of language and the information it carries rather than from joint training, and this points to practical future directions including model stitching, merging, and modular multilingual systems built from monolingual components.
Ej Zhou, Suchir Salhan, Catherine Arnett et al.· 1 citation
Cross-lingual contrastive learning has been a core component of multilingual encoder training, but the ability to explicitly align representations is not possible in decoder-only LLMs because of varying multilingual tokenization. However, a growing amount of research suggests that even in LLMs, higher cross-lingual rep...
Lucas Bandarkar, C. Peng, A. H. Ahmed et al.· 0 citations
(English) Machine Reading Comprehension (MRC) is a fundamental benchmark for evaluating natural language understanding systems, requiring models to generate answers grounded in context. Despite advances with transformer-based architectures and large language models (LLMs), challenges persist in multilingual and cross-l...
CTFAlign is introduced, a lightweight, training-free approach for document-level word alignment that applies a coarse-to-fine refinement strategy that restricts the alignment search space to semantically similar regions and introduces MDPAlign, a simpler alternative that constrains alignments by position with a main di...
Surprisingly, later-stage supervised fine-tuning on short texts further improves alignment quality while retaining the long-context understanding acquired in earlier training, keeping the model robust on long-text word alignment.
Mengpeng Yang, Jingxu Yang, Chao Chen et al.· 1 citation
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduNov 19, 2026
An MIT and Dana-Farber Institute-led team used large language models to examine nearly 4,000 studies and hundreds of code repositories, revealing gaps in the information needed to scrutinize and reuse published research.
By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.