Evaluating the Linguistic Properties of AI-Generated Bulgarian News Texts

Abstract

The rapid development of large language models (LLMs) has enabled the automatic generation of news articles that increasingly resemble texts written by human journalists. While a growing body of research has examined the linguistic properties of AI-generated texts in widely studied languages such as English, comparatively little work has focused on smaller languages such as Bulgarian. This raises important questions about how accurately contemporary language models reproduce the linguistic conventions of Bulgarian journalistic discourse. This paper presents a comparative evaluation of Bulgarian news texts generated by several large language models and authentic news articles written by human authors. The goal of the study is to assess the linguistic properties of AI-generated news and to identify systematic similarities and differences between machine-generated and human-written texts. To address this task, the paper applies a multi-level analytical framework combining semantic, stylistic and morphosyntactic analysis. The dataset consists of original Bulgarian news articles and texts generated by language models using prompts designed to produce news content on comparable topics. The analysis includes measures of semantic similarity between original and generated texts, as well as quantitative indicators describing lexical richness, stylistic patterns and structural characteristics of the texts. Morphosyntactic features are examined using automatic linguistic annotation tools for South Slavic languages, and statistical methods are applied to identify patterns across models and text types. The study contributes to ongoing research on the evaluation of AI-generated texts and provides a framework for analyzing the linguistic properties of machine-generated content in relatively low-resource languages such as Bulgarian.