The Context Tax of General-Purpose Tokenization

Abstract

This paper quantifies the context tax imposed by general-purpose tokenization in agglutinative languages. Using the same local Byte Pair Encoding (BPE) recipe, we train 64k native tokenizers for Turkish, Finnish, and Hungarian and compare them with tokenizers of GPT-4, mBERT, and XLM-R models. Each native tokenizer is trained on 30,000 Wiki40B texts and evaluated on a disjoint 500-text test set. The main metric is fertility, the average number of subword tokens per word; we also convert it into approximate word coverage under a fixed 512-token budget. In all three languages, the native tokenizer produces the shortest sequences. Native-vs-GPT-4 fertility gains are 77.63% for Turkish, 64.42% for Finnish, and 79.21% for Hungarian. The results show that tokenizer choice is a first-order context-efficiency decision in agglutinative languages.