Sequential Pretraining Favors Large Models
It is shown that small models can allocate learning capacity inefficiently toward early distributions, whereas sufficiently overparameterized models are robust to this effect, and Exposure Therapy (ET), a simple regularization that promotes more efficient allocation of learning capacity during sequential pretraining, i...