Token Ratios of the European Union's Official Languages Across Nine 2026 Tokenizers, and an Exploratory Tokenizer Transplant

Abstract

Language model services count, limit and bill text in tokens, but the number of tokens needed for the same content depends on the language and on the tokenizer. We measure this token ratio relative to English for the 23 non-English official languages of the European Union on nine tokenizers of 2026: six company tokenizers run locally (OpenAI o200k, Meta Llama 3.1, Alibaba Qwen3, DeepSeek V3, Google Gemma 3, Mistral Tekken), Anthropic's Claude generation-5 tokenizer measured through its counting endpoint, and two European research tokenizers (BSC Salamandra, EuroLLM). On NTREX-128 news, mean ratios range from 1.18 (Salamandra) to 1.98 (Qwen3). Text length does not suffice to explain the gaps: Czech and Estonian are no longer than English in characters, yet need 1.40 to 2.01 times the English tokens on the seven company tokenizers. On the lowest-count company tokenizer for each language, Greek needs 2.14 times the English tokens, and Latvian, Irish and Maltese 1.81 to 1.89. Salamandra needs 13% to 30% fewer tokens than that tokenizer for Central and Eastern European, Baltic, Greek and Nordic languages, while differences for Spanish, German, Italian, Portuguese and Dutch remain within about 4% and French requires 7% more tokens; EuroLLM saves at most 15%. The observed order of the eight locally run tokenizers is identical on FLORES+ and on the Universal Declaration of Human Rights. Claude's tokenizer gives the highest absolute count (3.08M tokens vs 1.87M for OpenAI). Finally, in an exploratory test we transplant a French-optimised tokenizer onto Qwen3-0.6B-Base: after one pass over 11 MB of professional French text, the transplanted models improve on the base model but remain behind a control adapted with its original tokenizer. These are segmentation results and one small adaptation experiment; effects on cost per task and on downstream task quality are not measured. Counts, scripts and transplant results: https://github.com/Baracoda-ai-labs/baracoda-fr. Follow-up to: T. Serval, The Invisible Language Tax, arXiv:2609.39001.