Knowledge Acquisition During Pre-training? Large Language Models Learn Better With Auxiliary Views
Gaps remain in our understanding of how large language models (LLMs) acquire knowledge during pre-training. We posit that auxiliary views, reformulations of knowledge, are causally helpful for learning. We design controlled experiments to isolate this. First, we confirm that repetition is necessary for acquisition and...
Joseph Lee, Yi-Di Huang, Dokyoon Kim et al.
· 0 citations