This study explores the semantic competence of Italian BabyLMs, focusing on their sensitivity to semantic violations, using a minimal pair benchmark targeting semantic violations to evaluate the semantic abilities of BAMBI, a family of small-scale models trained on progressively larger and more complex datasets.
Large Language Models (LLMs) are increasingly used for translation, yet their value depends on preserving meaning rather than producing fluent output. This study evaluates seven LLMs on Japanese–Croatian translation, a low-resource, typologically distant language pair. Using rubric-based human evaluation of adequacy, f...
In this paper, we draw a comparison between linguists in training, a trained linguist, and annotations generated by large language models (LLMs) to find out if they struggle with complex linguistic phenomena in a similar way. For this purpose, we analyse evaluative language in spoken popular science discourse, with the...
The results show that several widely used normalized metrics introduce crosslinguistic biases rooted in tokenization, encoding, and orthographic differences, and sentence-level negative log-likelihood computed over semantically equivalent sequences provides more meaningful and consistent crosslingual comparisons.
Xiulin Yang, E. Wilcox, Catherine Arnett· 3 citations
MORFES (Morphological Open-class Recognition-and-Formation Evaluation Suite), a benchmark of 500 expert-verified items that tests the recognition and production of Greek inflected forms, favoring lower-frequency lemmas so that a correct answer reflects the rule rather than a memorized form.
Abstract-Recently, large language models (LLMs) have enabled significant advances across a wide range of domains. However, the training, adaptation, and evaluation of LLMs with a large number of parameters incur substantial computational and financial costs. This limits the ability of researchers with constrained resou...
Osman Büyük, Mustafa Erden· Signal Processing and Commun...· 0 citations
This work presents the first systematic study of inverse relation directionality in LLMs, using a benchmark consisting of 5,457 instances spanning 27 distinct inverse relation labels and reveals systematic asymmetries in inverse relation classification across LLMs.
Sefika Efeoglu, Adrian Paschke· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
