Gradient Heterogeneity Complements Hessian Heterogeneity in Transformer Optimization
This study provides a theoretical analysis showing that gradient heterogeneity, together with Hessian heterogeneity, degrades the convergence of gradient-based methods such as SGD, while sign-based methods are substantially less sensitive to this effect.
Akiyoshi Tomihari, Issei Sato
· 6 citations
· ⚡1