You Only Need 2/3 of the Chosen Experts: An Empirical Study of Dynamic Expert Pruning in Fine-Grained MoE LLMs
It is found that expert selection is far more redundant than the field's operating points assume, and how much expert computation fine-grained MoEs can dispense with is revealed, and when dynamic allocation earns its complexity is established, informing both practical deployment and future pruning methods.
Yuan-Teng Chen, Qi-Wei Lai, Tian-Qi Chen et al.
· 1 citation