Optimizer-dependent training dynamics converge to the same one-third optimal data scaling
Neural scaling, in which loss falls as a power law with training, is central to large language models, and one recent proposal is that a $1/3$ exponent emerges from learning peaked distributions. That account describes SGD, but models in practice are trained with adaptive optimizers. Here we separate two exponents the...
Hyunseok Lee, M. Basil, Yizhou Liu et al.
· 0 citations