miLLi: Model Integrating Local Linguistic Insights for Morphologically Robust Tokenization
Standard statistical tokenization algorithms often struggle to preserve the morphological boundaries of agglutinative languages such as Azerbaijani. This study introduces miLLi 1.0, a hybrid tokenizer that integrates a rule-based root dictionary with a statistical Byte-Pair Encoding (BPE) approach. The model’s distingu...