Adam under Generalized Smoothness with Second-Moment-Type Stochastic Gradients
Adam is widely observed to remain stable even when the objective deviates significantly from global smoothness. Under the generalized smoothness framework, however, existing analyses rely on strong tail assumptions on the stochastic gradients, such as almost-sure boundedness or sub-Gaussianity. Whether Adam converges o...
Rui-Nan Jin, Di-Fei Cheng, Ling Chen et al.
· 0 citations