Towards a Statistical Understanding of Mixture-of-Experts
Mixture-of-experts (MoE) architectures increase model capacity by combining a collection of expert predictors through input-dependent routing, while often activating only a small subset of experts for each input. Despite their growing importance in modern large-scale models, the statistical roles of their design choice...
Si-Yuan He, Bo-Kai Yang, Jie Hu et al.
· 0 citations