COILD, an Indic-centric parallel corpus comprising over 1.16 million human-translated and human-verified sentence pairs, covers 20 Indian language pairs across the Indo-Aryan, Dravidian, Tibeto-Burman, and Austro-Asiatic language families, and provides a valuable training and evaluation resource for advancing multilingual machine translation and future multilingual language models for Indian languages.
Abstract
Machine translation (MT) for Indian languages remains constrained by the limited availability of high-quality, Indic-centric parallel corpora and evaluation benchmarks. Existing multilingual resources are largely constructed from English-pivot content and often fail to capture the linguistic diversity, cultural complexity, and domain-specific characteristics of Indian languages. We present COILD, an Indic-centric parallel corpus comprising over 1.16 million human-translated and human-verified sentence pairs, covering 20 Indian language pairs across the Indo-Aryan, Dravidian, Tibeto-Burman, and Austro-Asiatic language families. The corpus is built entirely from original Indian language sources collected from licensed repositories spanning eight domains with direct real-world applicability. Furthermore, we introduce a domain-centric benchmark comprising 2,000 expert-verified sentences to enable consistent multilingual and cross-lingual evaluation across Indian language pairs. To validate the effectiveness of COILD, we fine-tune two representative multilingual neural machine translation models, IndicTrans2-Distilled and NLLB-200. Experimental results demonstrate consistent improvements across language pairs, domains, automatic evaluation metrics, and human evaluation, highlighting the effectiveness of high-quality Indic-centric supervision. COILD provides a valuable training and evaluation resource for advancing multilingual machine translation and future multilingual language models for Indian languages.
A detailed comparative analysis of five state-of-the-art models in a zero-shot translation paradigm between English and four typologically different languages, Indic languages: Gujarati, Malayalam, Hindi, and Tamil indicates that there is significant performance difference among language pairs, translation directions a...
Vishal Chhabra, Atul AgrawaL· Journal of Intelligent Decis...· 0 citations
This paper analyses various recent state-of-the-art variants of large language models (LLMs) and neural machine translation (NMT) for Indian languages in comparison to statistical machine translation (SMT) and tackles key questions, such as idiomatic expressions, morphologically complex grammar or the scarceness of par...
Jayanand A. Kamble, Shivajirao M. Jadhav, V. J. Kadam· International Journal of Inf...· 0 citations
This work introduces the first-of-its-kind English-Assamese parallel corpus for the translation of Indian court judgments and performs a comprehensive evaluation of state-of-the-art multilingual models in both zero-shot and fine-tuned settings, comparing their performance against commercial systems.
Telem Joyson Singh, Hemanta Baruah, Sanasam Ranbir Singh et al.· International Conference on...· 1 citation
EnSiTa is presented, a trilingual multi-domain parallel dataset and benchmark for English, Sinhala and Tamil, and is the most extensive systematically documented multi-domain parallel data creation and benchmarking effort for low-resource MT.
Surangika Ranathunga, Nisansa de Silva, Aloka Fernando et al.· 1 citation· ⚡1
Machine translation offers speed and cost advantages, but human translation remains superior in quality and contextual understanding. For low-resource languages like Hausa, the severe scarcity of parallel data leads to poor model performance, overfitting, and inadequate handling of named entities. While existing synthe...
Rabiu Ibrahim Abdullahi, G. Obunadike, A. Abdullahi et al.· FUDMA Journal of Sciences· 0 citations
(English) Machine Reading Comprehension (MRC) is a fundamental benchmark for evaluating natural language understanding systems, requiring models to generate answers grounded in context. Despite advances with transformer-based architectures and large language models (LLMs), challenges persist in multilingual and cross-l...
C. Carrino· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduNov 19, 2026
An MIT and Dana-Farber Institute-led team used large language models to examine nearly 4,000 studies and hundreds of code repositories, revealing gaps in the information needed to scrutinize and reuse published research.
By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.