Evidence for Shared Routing Geometry and Dynamics in Sparse Mixture-of-Experts
Evidence is provided that routing-relevant states across layers share a common geometric structure that is obscured by layer-specific coordinate systems, and whether this shared dynamics is specific to routing or simply reflects the smooth evolution of hidden representations.
Kirill Labzin, Stepan Kulibaba, Artem Dzhalilov et al.
· 1 citation