Dataset Curation for Kalabari NMT System
TL;DR
A transferable curation framework for endangered languages, the first sizable Kalabari-English parallel corpus, and baseline experiments that reveal both the promise and the hallucination pitfalls of training on highly constrained, domain-specific data are contributed.
Abstract
The development of Natural Language Processing (NLP) tools for endangered and low resource languages is fundamentally hindered by the scarcity of high-quality parallel data. Data for languages like Kalabari is not only scarce but often noisy, inconsistently digitized, and orthographically unstandardized. While prior work has leveraged religious texts for corpus creation, the specific challenges of extracting and normalizing morphologically rich languages with complex diacritics remain underexplored. This paper addresses this gap by introducing a reproducible, modular curation methodology tailored for such languages. We document a six-step pipeline that transforms raw digital texts—sourced from the Kalabari Bible (FiaFia Biabulu) and instructional literature (Kalabari Lingua)—into a clean, verse aligned, 10,222-pair parallel corpus. We demonstrate that enforcing Normalization Form C (NFC) is critical for preserving sub-dot diacritics, and we validate the corpus by training a baseline Transformer NMT system. Our contributions are threefold: (1) a transferable curation framework for endangered languages, (2) the first sizable Kalabari-English parallel corpus, and (3) baseline experiments that reveal both the promise and the hallucination pitfalls of training on highly constrained, domain-specific data.