Only ILO's correlation with cross-lingual transfer (Spearman's $\rho = 0.90$) survives controls for model size, family, and per-task variation and is recommended as the primary sharing metric to be reported alongside anisotropy diagnostics.
Abstract
Multilingual language models develop shared cross-lingual representations, and various interpretability methods claim to quantify this sharing. These methods have been developed largely in isolation, and when they disagree, it is unclear whether the disagreement reflects a property of the model or an artifact of the measurement. We compare four sharing metrics (CKA, ANC, GMM dominance per token, and ILO) across 21 base models from five families (125M-14B parameters) and correlate each with cross-lingual transfer on five downstream tasks. We find that the metrics differ in their quantification of cross-lingual sharing in these models and suggest that the disagreement traces to anisotropy, the tendency of representations to cluster in a narrow cone of the embedding space. Only ILO's correlation with cross-lingual transfer (Spearman's $\rho = 0.90$) survives controls for model size, family, and per-task variation. We therefore recommend ILO as the primary sharing metric, to be reported alongside anisotropy diagnostics.
A controlled evaluation framework for multilingual natural language inference that stress-tests semantic consistency using synthetic logic-based premise–hypothesis pairs in both monolingual and code-switched conditions is presented.
This work presents a unified evaluation of representative CLC-enhancement methods for question answering, spanning inference-time interventions and post-training approaches across three model families and three closed-form benchmarks, highlighting the need to evaluate CLC enhancement for both cross-domain robustness an...
Jirui Qi, Ming-Yang Wang, Hinrich Schütze et al.· 0 citations
This work finds that identification probes systematically disagree: the GMM-based representation probe, which draws evidence from hidden state geometry, shows earlier cross-lingual mixing, whereas decoding-based probes, which rely on output-space decodability, retain sharper language-specific and more English-biased si...
Deniz Bayazit, Badr AlKhamissi, Antoine Bosselut· 0 citations
Large Language Models (LLMs) have made remarkable progress in the processing and modeling of many languages. Yet, unlike human multilinguals, they exhibit surprisingly limited cross-lingual knowledge transfer. While this limitation is well documented, its origins during multilingual training remain unclear. We pretrain...
Adam Gaber, Uriel Dolev, Elisabeth Fittschen et al.· 0 citations
It is confirmed that cross-lingual alignment can emerge from the structure of language and the information it carries rather than from joint training, and this points to practical future directions including model stitching, merging, and modular multilingual systems built from monolingual components.
Ej Zhou, Suchir Salhan, Catherine Arnett et al.· 1 citation
This work investigates how decoder-only transformers resolve lexical ambiguity through layer-by-layer analysis of three models spanning three parameter sizes, finding that representations become maximally distinct in middle layers, then partially reconverge in late layers, while the KL divergence between their next-tok...
K. Scott, Narun Pat, Veronica Liesaputra· 1 citation
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduNov 20, 2026
Senior Research Scientist Jim Glass discusses how his MIT-IBM Computing Research Lab collaboration connects perception and language, and addresses the shortcoming of current models.
MIT News · Artificial Intelligence· news.mit.eduNov 19, 2026
An MIT and Dana-Farber Institute-led team used large language models to examine nearly 4,000 studies and hundreds of code repositories, revealing gaps in the information needed to scrutinize and reuse published research.
By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.