A black-box, inference-time diagnostic that tells these two cases apart without retraining or annotation is introduced, with the pattern holding on English-Marathi and English-Tamil, with the failure modes tracking the post-edit distribution rather than MT quality or language family.
Abstract
Automatic Post-Editing (APE) for low-resource languages (LRLs) often fails to improve Machine Translation (MT), and the score alone cannot say why: whether more training would help, or whether the training data is too inconsistent to learn from. We introduce a black-box, inference-time diagnostic that tells these two cases apart without retraining or annotation. It varies an edit-distance penalty $\lambda$ that drives the model from free editing towards copying the MT, and reads two signals: (1) the shape of the Translation Edit Rate (TER)-vs-$\lambda$ curve, U-shaped if edits from the model reduce error and monotonically decreasing if none does; and (2) the ordering of constraint variants that trust model confidence to increasing degrees, which shows whether confidence tracks edit quality. Across decoder-only and encoder-decoder models on English-Sinhala, the diagnostic exposes two failure modes consistent with a heterogeneous post-edit signal as the underlying cause: Binary Collapse, where the model copies the MT or makes off-target edits, and Confident Miscalibration, where the confidence signals we test do not separate useful edits from unnecessary ones. The pattern holds on English-Marathi and English-Tamil, with the failure modes tracking the post-edit distribution rather than MT quality or language family. Beyond diagnosis, the curve shape prescribes a concrete next step for practitioners; in the favorable case, a static constraint yields a free inference-time accuracy gain. We release the first English-Sinhala (~66k) and a new English-Tamil (~39k) APE datasets with all code.
A regression-guided routing approach that prioritizes segments by predicted CER improvement, paired with a safeguard layer that detects harmful LLM corrections and routes uncertain segments to human review, and substantially outperforms standard confidence-based approaches is introduced.
Stergios Konstantinidis, Hayman Lotfy, Michalis Vlachos· Proceedings of the 2026 ACM...· 0 citations
The WMT 2020–2024 shared-task lineage with an extended English–Malayalam resource is consolidated into INDICQE-APE, with up to four label types aligned on the same segment, a direct assessment, a human post-edit, word-level OK/BAD tags and an error explanation, and a test set stratified over four difficulty axes.
Diptesh Kanojia, Archchana Sindhujan, S. Deoghare et al.· 0 citations
The WMT26 General MT task evaluates systems on 10 language pairs that have no human references (neither translated from scratch nor post-edited from MT output by humans) and how the pseudo-references were built is described.
Diptesh Kanojia, Chi-Kiu Lo, Archchana Sindhujan et al.· 0 citations
This work proposes augmenting existing benchmarks to increase translation difficulty by combining adversarial optimization with a differentiable translation difficulty estimator, and uses gradients from a combined difficulty and fluency objective to iteratively replace tokens in Adversarial Translation Optimization (AT...
William Kalikman, Šimon Sukup, Michal Tesnar et al.· European Association for Mac...· 2 citations
It is found that, in this benchmark, supervised fine-tuning (SFT) provides a strong baseline, substantially improving argument language consistency and end-to-end function call accuracy and, under consistent model selection, SFT achieves performance comparable to, and sometimes exceeding more complex reinforcement lear...
Siddharth Chauhan, Thomas Butler, Abhishek Singhania et al.· 0 citations
Desired-Update-Aligned Synthetic Data (DASA), which uses activation-gradient feedback from a frozen reference model to guide the optimization of continuous synthetic input embeddings, inspired by the role of activation gradients in local risk reduction, targets useful adaptation updates rather than source-text reconstr...
Jin-Hao Zhang, Ze-Yu Liu, Zi-Cheng Yan et al.· arXiv.org· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduNov 20, 2026
Senior Research Scientist Jim Glass discusses how his MIT-IBM Computing Research Lab collaboration connects perception and language, and addresses the shortcoming of current models.
MIT News · Artificial Intelligence· news.mit.eduNov 19, 2026
An MIT and Dana-Farber Institute-led team used large language models to examine nearly 4,000 studies and hundreds of code repositories, revealing gaps in the information needed to scrutinize and reuse published research.
By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.