A PMI-based translation metric is proposed, which is less dependent on the target language and correlates strongly with chrF, and finds that CLA with English predicts translation quality comparably to or better than source-target CLA.
Abstract
Multilingual large language models (LLMs) have been shown to perform better on non-English classification tasks when the representations of the given language are more aligned to English within the model. Several cross-lingual alignment (CLA) scores have been proposed for use with LLMs, along with multiple approaches for extracting embeddings from the models. We provide a comparative analysis of 27 CLA score variants, examining how they differ and how well each predicts downstream performance across three tasks. Crucially, while LLMs are widely used for generative tasks such as machine translation, prior work has focused almost exclusively on classification. We therefore investigate whether CLA scores are similarly predictive of translation performance. To enable computing correlations across target languages, we propose a PMI-based translation metric, which is less dependent on the target language and correlates strongly with chrF. We find that CLA with English predicts translation quality comparably to or better than source-target CLA, providing new evidence that LLMs use English as an internal pivot language.
This work introduces M-GATE (Multilingual Grammar, Accuracy in Translation, and Efficiency), a benchmark of linguistic proficiency spanning 30 typologically diverse languages from high- to low-resource, and evaluates over 50 models in more than 80 configurations.
Tomáš Burkert, Angelika Peljak-Łapińska, David Zelený· 0 citations
MTEB-PT is presented, a Portuguese benchmark constructed from a subset of MMTEB, comprising 14 existing datasets across Semantic Textual Similarity (STS), classification, retrieval, and reranking, and shows that language-specific fine-tuning still improves model performance in Portuguese, especially on task types that match the adaptation data most closely.
Lucas H.T. Okamura, Alexandre Alcoforado, A. H. R. Costa· 1 citation
Evaluation of multilingual large language models has grown rapidly in recent years, yet Marathi, spoken by over 83 million people across India, has received almost no systematic probing beyond surface-level benchmark tests. Most existing multilingual evaluations either omit Marathi entirely or rely on machine-translated test sets that fail to capture the morphological complexity that de-fines the language. We evaluate four models, namely Llama-3.1-8B, Llama-3.3-70B, Mistral-7B, and Qwen3-32B, on our manually curated Marathi dataset across three probing dimensions: Devanagari versus Roman-ized script, Marathi-English code-mixing, and syntactic structures including SOV word order, vibhakti case markers, verb gender agreement, and postpositions. Models are tested under English and Marathi instruction conditions across translation, similarity, grammat-icality, and case marker tasks. Translation quality is evaluated using both token-level F1 and BERTScore to capture paraphrase equivalence beyond surface word overlap. All models drop between 7.9% and 20.5% on Roman-ized input. The negative subjunctive marker nasta is ignored by every model. Vibhakti case markers are consistently replaced with Hindi equivalents, revealing that multilingual training has not produced separate internal representations for Hindi and Marathi despite their distinct morphological systems. These findings reveal structural gaps in how current multilingual LLMs handle morphologically rich, low-resource Indic languages and point to spe-cific areas where dedicated Marathi pretraining data would most benefit future work.
Tejas M Patil, Barnali Chetia· Proceedings of the 1st Works...· 0 citations
The results indicate that a moderately sized, shared self-attention architecture can deliver production-quality multilin-gual translation within the resource constraints of an academic de-ployment, while surfacing clear directions – low-resource language coverage, domain adaptation, and speech-based extension – for con-tinued development.
Darshan Gowda D H and Dr. Kruti R· International Journal of Adv...· 0 citations
Machine translation (MT) technologies are currently undergoing a paradigm shift, transitioning
from specialized Neural Machine Translation (NMT) frameworks to the broader capabilities of
Large Language Models (LLMs). This paper examines the current standing of the Croatian language
within this technological evolution.
While bilingual NMT models often exhibit high precision, multilingual NMT leverage transfer
learning to enhance performance for low–resource language pairs, but with lower performance for
high–resource ones. Conversely, LLMs—whether general–purpose or fine–tuned for translation—
offer superior multilingual proficiency and context awareness. Unlike NMT, LLMs can process extended discourse, such as full paragraphs or documents, leading to significant improvements in
coreference resolution and gender agreement. Despite the substantial computational requirements
of LLMs, recent optimization techniques allow for smaller, more efficient versions that maintain
high output quality.
This study evaluates the performance of various NMT and LLM architectures specifically for
Croatian from/to English and Spanish using several automatic quality evaluation metrics. The findings demonstrate that open–source models can achieve, and occasionally surpass, the quality of
Google Translate, a widely used commercial NMT system. Furthermore, while our evaluation focuses on this specific language triad, the multilingual nature of the analysed systems suggests that
open–source models provide high–quality translation capabilities for Croatian across dozens, if not
hundreds, of language pairs.
A. Oliver, Sergi Álvarez–Vidal· Suvremena Lingvistika· 1 citation
In every setting, pre-adaptation on related auxiliary languages yields no practically meaningful improvements once as little as one hour of target-language data is available, suggesting that relatedness alone may not reliably predict transfer gains in large multilingual ASR, or constitute an effective strategy for extending such models to low-resource languages.
A. Florian, C. Amol, Hope Kerubo Ombaba et al.· 0 citations