Preserving Morphemes: Morphology-Guided Pre-Tokenization for Nepali
Downstream the effect is small: NER improves only on entities that contain words unseen in training, POS tagging and news classification do not change, and published Nepali tokenizers perform about as well.
Kalash Shrestha, Nikhil Pradhan
· 0 citations