Standard statistical tokenization algorithms often struggle to preserve the morphological boundaries of agglutinative languages such as Azerbaijani. This study introduces miLLi 1.0, a hybrid tokenizer that integrates a rule-based root dictionary with a statistical Byte-Pair Encoding (BPE) approach. The model’s distinguishing feature is a dynamic phonological restoration algorithm designed to map allomorphic variations back to their canonical root forms. Empirical evaluations on the Tatoeba corpus demonstrate that miLLi 1.0 (with a 32,000 vocabulary size achieving 1.980 T/W) outperforms global standards such as GPT-4o and mBERT in terms of representation efficiency. While exhibiting lower token density compared to local statistical models, miLLi 1.0 demonstrates high linguistic robustness, achieving 58.0% in Morphological Boundary Accuracy (MBA) and 79.0% in Root Consistency Rate (RCR), significantly outperforming a standard BPE baseline trained under identical conditions (47.0% MBA and 1.0% RCR). The findings suggest that the integration of a linguistic filtration layer establishes an optimal balance between statistical compression and semantic integrity.
Results show that BPE produces the most compact and morphologically aligned subword representations, while the modified Unigram LM achieved the best overall downstream performance across tasks, underscore that tokenizer choice and vocabulary design are critical determinants of language model efficiency and performance...
Mátyás Osváth, Matej Molnár, Roland Gunics et al.· International Conference on...· 1 citation
Byte-level BPE tokenizers that use the HuggingFace ByteLevel pre-tokenizer inherit GPT-2's word regex, where a word is defined as \p{L}+, one or more Unicode letters, which splits each word at every vowel sign, and formalises this effect as a training-free lower bound on fertility.
S. Regmi, Siddhartha Pudasaini, Chetan Phakami Pun· 2 citations
CNeo-Bench, a benchmark of 4,759 Chinese neologisms with reference definitions, is introduced, organized into five top-level categories and nine subcategories by the linguistic mechanism behind each expression, paired with a two-tier evaluation framework that separates whether a model can describe a neologism from whet...
Kai-Yan Zhao, Zhong-Tao Miao, Zhe-Yong Xie et al.· 0 citations
Part-of-speech (POS) tagging in Arabic is hard because its rich root-and-pattern morphology and the absence of short vowels make one unvoweled string compatible with many categories. This paper presents Jawhar, a hybrid framework that couples a high-performance morphological analyser with contextual reranking using a p...
Mohamed Bouzahir, A. A. Abdelouahad, M. Nabil· Information· 0 citations
This paper investigates the morphological realization of tense and the syntactic checking of the tense feature in Standard Arabic (SA). The empirical point of departure is a distribution puzzle: SA possesses only two verbal forms, the perfective and the imperfective, yet expresses three temporal distinctions. A substan...
Adil Mrhar, Ahmed Smirkou, Nabil Guejouti et al.· International Journal of Lin...· 0 citations
This paper uses morphological feature annotations from the Universal Dependencies corpora to quantify information carried by morphology across 154 different datasets spanning 72 language varieties, and proposes an information-theoretic approach that measures how surprising morphological feature values are given a token...
Hedvig Skirgård, S. Mann· Linguistics Vanguard· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
