Root, Suffix, or Byte? A Reproducible Study of Sub Word Tokenization for a Morphologically Rich Language (Russian)

Abstract

Tokenization is the first design choice in a language model and one of the most frequently overlooked for morphologically rich languages, as it directly shapes sequence length and how related word forms are segmented into reusable sub word units. Using Russian as a case study, we train sub word tokenizers from scratch on a small, morphology-dense corpus and quantify two effects. First, a coverage penalty: a hypothetical English-centric byte-level tokenizer that has never learned Cyrillic falls back to raw UTF-8 bytes and spends 8.6× more tokens on the same Russian text than a Russian-trained Byte-Pair Encoding (BPE) tokenizer. Second, a compression morphology trade-off as vocabulary grows, BPE fertility falls toward one token per word, but it does so by memorizing complete inflected word forms rather than by isolating morphemes, so a nine-form paradigm of «дoм» (house) is stored as nine unrelated tokens with zero root sharing. A Unigram tokenizer at a comparable budget instead factors words into a shared root plus a shortening, representing the same paradigm with fewer unique token types at the price of slightly higher fertility. We release the corpus, training scripts, and an in-browser reimplementation of the trained BPE model that reproduces the reference tokenizer exactly. The numbers are illustrative, but the mechanisms are real and match findings in the recent Russian NLP literature.