Jul 2026
In-Place Tokenizer Expansion for Pre-trained LLMs
A tokenizer fixed at the start of pre-training allocates vocabulary in proportion to the pre-training corpus, reflecting the deployment priorities at that time. When those priorities shift, languages added later are split into many more tokens per word, which can raise latency, compute, and energy consumption for users...
Jimmy T.H. Smith, Tarek Dakhran, Alberto Cabrera et al.
· arXiv.org · 0 citations