Hengshuai Yao

Training-Free Halving of Activated Experts in Fine-Grained Mixture-of-Experts Models

Modern fine-grained Mixture-of-Experts (MoE) models route each token to a small number of experts and renormalize their router probabilities. We show that this renormalization implicitly calibrates expert output gain to the training top-$k$: reducing $k$ at inference changes not only which experts are used but also the...