3 papers indexed here
We haven’t gathered this author’s papers yet. Follow them and we’ll fetch their work.
Not the right person? Other researchers publish under this name.
The theory yields simple dynamic momentum selection rules for both SGD+Momentum and Adam/AdamW, and shows that the resulting dynamic momentum rules match or improve upon the best fixed hyperparameters found via manual sweeps, reducing the need for exhaustive momentum sweeps.
This work derives the edge of stability threshold from the directional Hessian and the gradient-alignment score with respect to the actual update taken by the optimizer, rather than the maximum curvature mode.
It has recently been found that deep learning often occurs at the"edge of stability (EoS),"where the maximum Hessian eigenvalue of the model is stabilized at a value reciprocal to the learning rate. However, what happens before we reach that regime? We fix a deep learning problem and vary first order optimization metho...
We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.
