Descriptive adequacy and Large Language Models

Abstract

Generative language models produce unconstrained text with few grammatical errors. But full linguistic competence also requires the ability to associate grammatical sentences with the correct syntactic structures, resulting in the correct semantic interpretations. We performed experiments to establish whether five commercially available LLMs are able to distinguish structurally ambiguous sentences from unambiguous ones in Dutch, and to assign the correct interpretations to the unambiguous variants. In order to minimize the effects of pragmatics, an additional experiment was performed with stimuli containing pseudowords. The results are compared with results obtained from human subjects. It is found that the LLMs tested indeed show an ability to recognize structural ambiguity and to choose the correct semantic interpretations, although they perform worse than humans, and show different biases.