Lisa Bylinina

Augustinian BabyLM: What Ostensive Definition Can and Cannot Teach a Small Language Model

A language model normally begins training with random word embeddings: whatever'banana'means must be learned from training corpora. I implement St. Augustine's picture of word learning, meaning by ostension, for a small masked language model (DeBERTa) trained on 10M words: before training, visually grounded tokens rece...