Designing Chinese Tokenizers under Data Constraints: A White-Box Empirical Study of Character Fragmentation

Abstract

Chinese text requires substantially more tokens than English in large language models (LLMs), increasing API costs and context-window consumption. Most prior work measures this premium using off-the-shelf tokenizers, but the underlying mechanism—where the additional tokens actually arise—remains poorly understood, particularly under data-constrained training, which is common in domain-specific and small-scale models. We implement BPE, WordPiece, and Unigram from scratch (approximately 1,000 lines of standard-library code), train them on 20,000 Chinese news documents, and quantify per-character tokenization across vocabulary sizes from 8K to 128K. Characters are grouped by frequency, and evaluation uses a held-out test set with clustered-bootstrap confidence intervals. The central result is a low-frequency fragmentation effect: at an 8K vocabulary, byte-level BPE splits characters appearing fewer than 100 times in training into an average of 2.04 tokens per character, compared with 1.00 for high-frequency characters; enlarging the vocabulary 16-fold reduces this value to 1.10 at a fixed training scale. Comparison with production tokenizers reveals a spectrum from 2.24 (OpenAI's cl100k) and 2.04 (our 8K implementation) to 1.02 (DeepSeek's Chinese-optimized 128K vocabulary), indicating that the effect depends on vocabulary allocation and training scale rather than being an inherent property of Chinese. A semantic-equivalence experiment further supports this conclusion: for equivalent Chinese-English sentence pairs, an English-centric tokenizer uses 65% more tokens for Chinese, whereas a balanced byte-level BPE trained on 20,000 documents (10K Chinese + 10K English) uses 27% fewer. The paper concludes with configuration guidance for Chinese-focused tokenizers under data constraints.