Router Prior Bias: Preserving Base Routing Structure in MoE Post-Training
Mixture-of-Experts (MoE) pretraining relies on an auxiliary load-balancing loss (LBL) to drive per-expert utilization toward uniformity. Post-training inherits a different situation: the base router already encodes non-uniform expert co-activation structure, which a re-imposed uniformity objective flattens away. We sho...
Jaedeok Lee, Keonwoo Kim, Dongyoon Han et al.
· 0 citations