A Quantitative Framework for Measuring Linguistic Domain Gap in Indonesian Government Language Models

Abstract

Domain-specific pre-trained language models (PTLMs) for the Indonesian government sector, such as IndoGovBERT, are typically trained on national-level corpora yet deployed on sub-national administrative text with markedly different linguistic characteristics, such as village development documents. The decision to apply Domain-Adaptive Pre-Training (DAPT) before fine-tuning is usually made heuristically, without an explicit measurement of the underlying linguistic domain gap. This study develops a quantitative, multi-metric framework for measuring linguistic domain gap between government sub-domains and applies it to compare a national-level source corpus with a newly constructed village-development target corpus. The results reveal a pronounced structural gap—canonical regulatory markers appear in over 97% of village documents versus under 40% nationally—alongside low vocabulary overlap (Jaccard = 0.011) and a meaningfully higher out-of-vocabulary rate for village text against IndoGovBERT’s own tokenizer (8.72% vs. 6.80%), jointly indicating a moderate, heterogeneous domain gap. These differences are confirmed statistically significant (two-proportion z-test, p < 0.0001 for all structural markers and the OOV rate), and this evidence is supported by explicit, pre-registered decision thresholds rather than post-hoc qualitative judgment. These findings support Light-DAPT as the recommended adaptation strategy for this corpus pair, offering evidence-based justification for DAPT ahead of costly pre-training investment.