Percolation Dynamics in Optimization: Variance Cascades and Nested Symmetry
We study the dynamics of Stochastic Gradient Descent (SGD), which is known to steer deep neural networks toward invariant sets that correspond to simpler subnetworks. How this steering unfolds over time remains poorly understood. We answer this by modeling the stochastic gradient flow (SGF) as a percolation process, in...
Sai Niranjan Ramachandran, Suvrit Sra
· 0 citations