SuTRA : Structurally-Unified Tokenization with Root Awareness
This work proposes SuTRA (Structurally-Unified Tokenization with Root Awareness), a morphology-aware algorithm that preserves akshara indivisibility and penalizes merges crossing morphological boundaries and releases a new morphological segmentation dataset for Hindi, Marathi, and Gujarati.
Vaibhav Rathore, Siddhant Gole, Dadhichi Telwadkar et al.
· 0 citations