Tokenization Explained: What It Is, Why It Matters, and How to Work Around Its Limits

Analysis of the Efficiency of Subword Tokenizers in a Low-Resource Linguistic Environment: Implementation Experience for the Tajik Language

Can Knowledge Be Translated (by a Machine)?

This paper concludes that machines may assist translators but that they will not, by principle, be able to reach an almost perfect level and object to the huge amount of money spent on software development for systems with that objective.

Detecting and correcting Factual Error in LLM Text series: Review

This thesis presents a detailed introduction to the research work. It begins by establishing the background and motivation for the study, focusing on the growing importance of automatic text summarization and the challenges associated with factual inconsistencies in generated summaries. The chapter critically reviews prior research in the field of text summarization, highlighting the limitations of traditional Automatic Text Summarization (ATS) systems, particularly in ensuring factual correctness. Furthermore, the chapter introduces Large Language Models (LLMs) and prompt engineering as emerging solutions capable of addressing these limitations. The objectives of the research, along with the key research questions, are clearly articulated to define the scope and direction of the study. The chapter also delineates the boundaries of the research by specifying the scope and assumptions considered. Finally, the significance of the study is discussed, emphasizing its contribution to improving evaluation methodologies for text summarization systems. . Key Words: Natural Language Processing (NLP), Large Language Models (LLM), Chain of Thought (COT), Generative Pre-trained Transformers (GPT), Recall-Oriented Understudy

The Context Tax of General-Purpose Tokenization

This paper quantifies the context tax imposed by general-purpose tokenization in agglutinative languages. Using the same local Byte Pair Encoding (BPE) recipe, we train 64k native tokenizers for Turkish, Finnish, and Hungarian and compare them with tokenizers of GPT-4, mBERT, and XLM-R models. Each native tokenizer is trained on 30,000 Wiki40B texts and evaluated on a disjoint 500-text test set. The main metric is fertility, the average number of subword tokens per word; we also convert it into approximate word coverage under a fixed 512-token budget. In all three languages, the native tokenizer produces the shortest sequences. Native-vs-GPT-4 fertility gains are 77.63% for Turkish, 64.42% for Finnish, and 79.21% for Hungarian. The results show that tokenizer choice is a first-order context-efficiency decision in agglutinative languages.

Attend to Fragments: How Key Information Affects Large Language Models for Factual Inconsistency Detection

A new benchmark, KIFI, is designed, which comprises 1032 carefully selected instances from the TRUE and ScreenEval datasets, with key information annotated, and it is shown that LLMs frequently fail to use the appropriate information to make correct decisions.

The Shortcomings of Natural Language Processing (NLP) Models and Their Applications in Achieving Accuracy in Translating from Arabic to English

Translation is defined as the process of transferring meaning from one language to another. It is an extremely difficult and complex process because it involves not only transferring words but also ideas, culture, linguistic customs, and meanings derived from syntactic elements, their arrangement, word structure, and derivation. This is especially true in languages with complex structures, such as Arabic, which is characterized by its multiple linguistic contexts. These contexts have not been adequately addressed by NLP (Natural Language Processing) applications in machine translation models due to the lack of diverse contexts where metaphor, figurative language, grammatical inflections, morphological patterns and their connotations, and sentence structure all play pivotal roles in determining meaning. Furthermore, spoken language, with its inherent phonetic and expressive characteristics, conveys the text into broader semantic spaces. These spaces are influenced by the effect of intonation on specific syllables, the speaker's psychological state, the listener's mood, and accompanying body language, which transforms meaning into other subtle details. All of this, and more, is absent from machine translation, no matter how hard its creators try to imbue it with human emotions and feelings. This study highlights the importance of integrating in-depth linguistic analysis, contextual semantic modelling, and cultural awareness into natural language processing-based translation systems. By combining traditional linguistic insights with computational methods, the research offers a framework that can contribute to improving the accuracy of machine translation from Arabic to English.