Pretraining Data Statistics Shape the Phases of Learning Entity Comparison in Language Models
It is demonstrated that properties of pretraining data are causal drivers of heuristic learning and show that small-scale synthetic experiments can predict training dynamics at larger scales.
Siu-Chan Yik, Jing Huang, Yanai Elazar et al.
· 1 citation