Jul 2026· Signal Processing and Communications Applications Conference· pp. 1-4· 0 citations· 9 references
Abstract
This paper quantifies the context tax imposed by general-purpose tokenization in agglutinative languages. Using the same local Byte Pair Encoding (BPE) recipe, we train 64k native tokenizers for Turkish, Finnish, and Hungarian and compare them with tokenizers of GPT-4, mBERT, and XLM-R models. Each native tokenizer is trained on 30,000 Wiki40B texts and evaluated on a disjoint 500-text test set. The main metric is fertility, the average number of subword tokens per word; we also convert it into approximate word coverage under a fixed 512-token budget. In all three languages, the native tokenizer produces the shortest sequences. Native-vs-GPT-4 fertility gains are 77.63% for Turkish, 64.42% for Finnish, and 79.21% for Hungarian. The results show that tokenizer choice is a first-order context-efficiency decision in agglutinative languages.
This work quantifies the tokenizer tax for Indian languages using the FLORES-200 parallel corpus, measuring tokenization fertility across six widely used tokenizers and fourteen languages, and identifies the primary mechanism behind this disparity: failed byte-pair merges that leave text fragmented into single-byte tok...
Byte-level BPE tokenizers that use the HuggingFace ByteLevel pre-tokenizer inherit GPT-2's word regex, where a word is defined as \p{L}+, one or more Unicode letters, which splits each word at every vowel sign, and formalises this effect as a training-free lower bound on fertility.
S. Regmi, Siddhartha Pudasaini, Chetan Phakami Pun· 2 citations
A token-cost ledger is assembled that splits each language's cost, at fixed parallel content, into a removable coding redundancy, a residual coding slack, an intrinsic-content term, and an orthogonal, irreducible grapheme-to-phoneme term that governs the multimodal rather than the text cost.
Modern multilingual tokenizers often fragment Ukrainian and other underrepresented Cyrillic-script languages more heavily than English, creating disparities in cost and context capacity. We quantify this overhead across nine production tokenizers and five languages with standardized Cyrillic and Latin representations,...
A tokenizer fixed at the start of pre-training allocates vocabulary in proportion to the pre-training corpus, reflecting the deployment priorities at that time. When those priorities shift, languages added later are split into many more tokens per word, which can raise latency, compute, and energy consumption for users...
Jimmy T.H. Smith, Tarek Dakhran, Alberto Cabrera et al.· arXiv.org· 0 citations
The experimental results revealed the strengths and weaknesses of various approaches to subword segmentation and identified the most effective tokenization strategies under the conditions of the morphological complexity of the Tajik language.
M. Arabov, S. S. Khaybullina· Automatic Documentation and...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
