Normalize-Then-Precondition: A Hierarchical Approach to Marginal Scale and Interaction Geometry for LLM Training
NormPre is developed with NormPre-G and NormPre-L adopting global and localized spectral preconditioning, grounded in spectral-norm steepest descent and a regularized formulation followed by leading mode selection, respectively, to enable large-scale training.
Zi-Xuan Gong, Ze-Yu Gan, Jia-Ye Teng et al.
· 0 citations