Jul 2026
Mini-batch Noise Lowers Sharpness via Dominant-Subspace Fluctuations
This work shows how the averaged gradient over fluctuations in the dominant directions produces a sharpness correction term, and derive a sharpness correction term induced by mini-batch noise in the dominant directions, which brings the sharpness evolution of GD closer to that of SGD.
Jun-Ho So, Dongwook Shin
· arXiv.org · 1 citation