Context-Sensitive N-Gram Word Partitioning for Improving the Quality of Turkish Word Embeddings
TL;DR
The proposed approach provides a language-agnostic, context-sensitive segmentation mechanism that can complement language processing methods such as lemmatization, morphological analysis, and stemming and indicate task-dependent and generally limited improvements over traditional token-based word-embedding extraction.
Abstract
Current advances in neural network models have improved state-of-the-art performance in natural language processing tasks such as named-entity recognition, sentiment analysis, and machine translation. In particular, neural language models are applied to encode information in word embeddings. These approaches are generally trained on large corpora using semi-supervised learning. Word embeddings encode the syntactic and semantic properties of words as dense vectors. In agglutinative languages such as Turkish, Finnish, and Hungarian, word-embedding construction is challenging because extensive suffixation and polysemy can cause information loss. To overcome these limitations, character n-grams are often preferred for embedding representations. Nevertheless, character n-grams do not guarantee the capture of information in long word sequences. In this study, a method that partitions word sequences according to frequent patterns within a given context is proposed for training a neural language model. In this respect, likelihood- and ranking-based inference are combined with n-gram and syllable partitioning for word-embedding generation from a text corpus. The proposed approach provides a language-agnostic, context-sensitive segmentation mechanism that can complement language processing methods such as lemmatization, morphological analysis, and stemming. For embedding generation, the SkipGram and FastText models are used, and the effects of word partitioning are evaluated using analogy, named-entity recognition, POS tagging, sentiment analysis, and morphological disambiguation datasets for Turkish. The results indicate task-dependent and generally limited improvements over traditional token-based word-embedding extraction. In particular, skip n-gram partitioning produces a substantial improvement over partitioning based on frequent-ngrams, sentencepiece-bpe, sentence-unigram and morfessor. No consistent relationship was observed across tasks between performance and either graph density or the average number of distinct n-grams per sentence.