Subword Tokenization for Low- and Medium-Resource Languages: A Systematic Evaluation
Evaluation of subword tokenization strategies in six diverse low- and medium-resource languages shows statistically significant improvements for part-of-speech tagging, named entity recognition, and question answering, but not for dependency parsing and summarization, and byte-level tokenization provided no measurable...