Skip to content
Open access

Subword Tokenization for Low- and Medium-Resource Languages: A Systematic Evaluation

Sep 2026 · IJCAI/ECAI · pp. 5684-5692 · 1 citation · 50 references
Computer Science

TL;DR

Evaluation of subword tokenization strategies in six diverse low- and medium-resource languages shows statistically significant improvements for part-of-speech tagging, named entity recognition, and question answering, but not for dependency parsing and summarization, and byte-level tokenization provided no measurable advantage in a monolingual setting.

Abstract

Subword tokenization is a standard technique for pre-trained language models, mapping text into sequences of tokens from a fixed-size vocabulary. Despite its widespread use, the impact of tokenization algorithms and vocabulary sizes on downstream performance remains underexplored, particularly for low- and medium-resource languages, where suboptimal configurations are difficult to compensate with additional data. Prior studies have mainly focused on high-resource languages, individual tokenization algorithms, or fixed vocabulary sizes, limiting their scope. In this paper, we present a systematic evaluation of subword tokenization strategies in six diverse low- and medium-resource languages: Icelandic, Estonian, Basque, Galician, Nepali, and Tajik. We pre-trained monolingual TEAMS-Small models using WordPiece, BPE, and Unigram tokenizers with vocabulary sizes of 16k, 32k, and 64k, and additionally compared byte-level and character-level tokenization for Icelandic. Evaluation on a benchmark of NLP tasks revealed that Unigram tokenization with a 64k vocabulary consistently outperformed other configurations, with vocabulary size having a greater impact on downstream performance than algorithm choice. These gains were task-dependent, with statistically significant improvements for part-of-speech tagging, named entity recognition, and question answering, but not for dependency parsing and summarization. Finally, byte-level tokenization provided no measurable advantage in a monolingual setting, suggesting that its benefits are primarily relevant for multilingual models.

Read PDF

Similar papers

#machine learning Preprint Nov 2026

MoTo: Mixture of Tokenizers Towards Fair Multilingual Language Modeling

Large language models rely on a single tokenizer that is chosen at training time and fixed thereafter. Most tokenizers are trained on English-dominant corpora, making them under-serve morphologically rich and non-Latin-script languages, where they can produce fragmented representations, longer sequence lengths, and red...

G. Altintas, Colin A. Raffel · 0 citations
#natural language process... Preprint Oct 2026

Counting and Min-Cost Encoding for Tokenization in Large Language Models

Mainstream large language models rely on a tokenizer to encode text into a token sequence. Different tokenizers may yield token sequences of substantially different lengths for the same text. With a fixed model architecture, shorter token sequences correspond to lower inference time. We propose a tokenizer training app...

Shu-Ming Shi, Xiang Zhang, Hao-Yi Yu et al. · 0 citations
Preprint Nov 2026

LATTE-WordSeg-Lite: Multi-Granularity Thai Word Segmentation on a CPU, with a Reproducible Benchmark

Large language models read Thai through subword tokens, without a word segmenter. Yet Thai treebanks, speech metrics and morphological resources are built on words, often at several granularities, and word boundaries resolve ambiguities that subword tokens leave open. LATTE (Lattice ATTentive Encoding) reads a text at...

Thodsaporn Chay-intr, Kobkrit Viriyayudhakorn, Thanaruk Theeramunkong · 0 citations
#natural language process... Preprint Oct 2026

More Than Words: Compositional Tokenization for Efficient Language Models

Language models process and generate text sequentially in token units, and the tokenizer determines how much text each inference step covers. Under standard tokenization, a short English phrase such as"On the table."is usually produced as four separate predictions for the preposition (On), article (the), noun (table),...

Yuval Reif, Guy Kaplan, Roy Schwartz · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.