Learning Orthogonal Multi-Index Models Beyond Small Initialization: Incremental Learning, Competitive Dynamics and Symmetry
This work proves that incremental learning still occurs, and introduces a symmetry-based finite-width approximation via symmetrized networks, rather than comparing directly with an infinite-width limit, which yields better control of approximation errors and may be of independent interest.