L. Gong

Designing Chinese Tokenizers under Data Constraints: A White-Box Empirical Study of Character Fragmentation

Chinese text requires substantially more tokens than English in large language models (LLMs), increasing API costs and context-window consumption. Most prior work measures this premium using off-the-shelf tokenizers, but the underlying mechanism—where the additional tokens actually arise—remains poorly understood, part...