Sharpness-aware Orthogonalized Momentum Algorithms
Adversarial sharpness penalty per iteration of a base optimizer, known as sharpness-aware minimization (SAM), is an accepted technique to improve generalization in deep learning. Recently, Muon has emerged as a dominant optimization algorithm for training large language models (LLMs). It achieves lower sample complexit...
Geetika, Shivani Singh Chauhan, Gauranshi Gupta et al.
· 0 citations