Jul 2026
BHARATI: Morphology-Aware Tokenizers for Classical Indian Languages with Subword Fertility Analysis
BHARATI is presented, a set of SentencePiece BPE tokenizers trained on a balanced 781 MB corpus spanning seven languages with native script support for all languages, and three successive tokenizer versions are described.
P. Kumaresan, P. Muruganantham, L. Rajendran et al.
· arXiv.org · 0 citations