Zi-Yin Liu

Neural Quadratic Forms: A Unified Minimal Model for Sudden Learning and Scaling Laws

It is shown that a symmetry fixes what variables are: a network layer is a sum over interchangeable units, so relabeling the units leaves it unchanged; given smoothness and the condition that a unit's gradient vanish at the origin, symmetry then enforces a universal leading form for the expansion about the near-zero weights present at the start of training.