How Does Local Landscape Geometry Evolve in Language Model Pre-Training?
The scale and expense of pre-training language models make efficient hyperparameter tuning essential, yet a principled guidance is still missing. In this work, we analyze language model pre-training dynamics from a local landscape geometry perspective. Our study reveals two distinct phases. In Phase I, sharpness of the...
Zhan-Peng Zhou, Yu-Han Sun, Bing-Rui Li et al.
· 1 citation