Memory-Efficient Expert Routing for Distributed MoE Training
This work proposes RelayMoE, a ring-based MoE execution model that computes locally as expert weights or tokens circulate, avoiding full top-$k$-expanded dispatch buffers and achieves a $2\times$ average speedup over Megatron-LM.
Arnab Kanti Tarafder, Jaume Guasch-Martí, Gokcen Kestor et al.
· 0 citations