A Data-free Universal Prior over Syntactic Structures

Descriptive adequacy and Large Language Models

Generative language models produce unconstrained text with few grammatical errors. But full linguistic competence also requires the ability to associate grammatical sentences with the correct syntactic structures, resulting in the correct semantic interpretations. We performed experiments to establish whether five comm...

Neural Language Models Learn the Contextual Distributions of Dependency Structures: a statistical learning theory to compositionality

It is argued that this process provides a statistical-learning account of how NLMs solve the compositionality problem in language, and the possibility that this statistical learning process provides an explanatory theory on how language cognition could emerge from pure distributional statistics.

A theory of platonic representations in language models

Representations of translated sentences are similar in the inner layers of multilingual language models -- an observation connected to the platonic representation hypothesis, yet unexplained theoretically. We provide an explanation based on the assumption that data have a hidden hierarchical structure whose abstract le...

The Structural Sources of Verb Meanings Revisited: Large Language Models Display Syntactic Bootstrapping

The results show that models’ verb representations degrade more when the authors perturb word order, which removes syntactic information, than when they perturb co-occurrence among content words, which removes semantic information.

Learning to Learn a Language

We present the Prior-Fitted Language Model (PFLM), a 300M-parameter byte-level transformer pretrained only on samples from a synthetic non-linguistic prior. Given a prefix of real text, it learns to predict the language in context with frozen weights, having never seen a word of any real language. Every training sequen...

Nawatl Context-Free Grammars for Natural Language Processing