BHARATI: Morphology-Aware Tokenizers for Classical Indian Languages with Subword Fertility Analysis
BHARATI is presented, a set of SentencePiece BPE tokenizers trained on a balanced 781 MB corpus spanning seven languages with native script support for all languages, and three successive tokenizer versions are described.