Mathis Immertreu

Word-Class and Construction-Like Structure Emerges in Neural Successor Representations Trained on Natural Language

Language models are typically trained to predict the next token in a sequence. Here, we explore an alternative predictive principle from reinforcement learning: Successor Representations (SRs), which model the expected discounted distribution of future states rather than the immediate next state. We transfer this frame...