Skip to content

Memory-Efficient Expert Routing for Distributed MoE Training

Oct 2026 · 0 citations · 43 references
Computer Science

TL;DR

This work proposes RelayMoE, a ring-based MoE execution model that computes locally as expert weights or tokens circulate, avoiding full top-$k$-expanded dispatch buffers and achieves a $2\times$ average speedup over Megatron-LM.

Abstract

As Mixture-of-Experts (MoE) models scale toward hundreds of experts and higher top-$k$ routing, memory efficiency in distributed training becomes a critical bottleneck. Peak memory is dominated by the MoE block, not attention: every intermediate buffer in the MoE dispatch pipeline is individually scaled by top-k routing. The standard all-to-all dispatcher sends all routed tokens in a single collective step, requiring the full top-$k$-expanded buffer to be constructed at once. In this work, we propose RelayMoE, a ring-based MoE execution model that computes locally as expert weights or tokens circulate, avoiding full top-$k$-expanded dispatch buffers. RelayMoE selects between expert and token routing according to communication volume and overlaps transfers with computation. The ring structure naturally supports memory-efficient MoE recomputation during backward: each hop reconstructs expert intermediates, uses them to compute gradients, and releases them before the next hop. The saved memory supports longer sequences and larger batches, or retains more attention activations to reduce attention recomputation and improve training throughput. We evaluate RelayMoE on 30B$-$57B production MoE models and varied expert configurations. In single-layer MoE experiments, RelayMoE achieves a $2\times$ average speedup over Megatron-LM. In full-model training under the same GPU memory budget, it improves throughput by up to $2.02\times$ and extends the largest tested trainable sequence length by up to $2.85\times$.

View source

Similar papers

Preprint Oct 2026

Accelerating MoE Training through Expert Placement and Replica Coordination

Distributed training of sparsely activated mixture-of-experts (MoE) models is often bottlenecked by workload imbalance and all-to-all (A2A) communication induced by dynamic token routing. Existing token deduplication, expert placement, and replication techniques alleviate these bottlenecks but do not jointly capture th...

Zhi-Qiang Tan, Mao-Xin Wang, Wen-Jing Lin et al. · 0 citations

TopoEP: Topology-Aware Load Balancing for Expert-Parallel MoE Training

A GPU-native, topology-aware load-balancing system for large-scale MoE training that converts the current routing result into hot-expert replication and token-rerouting decisions and executes the resulting plan without data-dependent host synchronization, reducing critical-path overhead.

Jia-Cheng Zhu, Xie Zhao, Gong-Ming Zhao et al. · 1 citation
Preprint Oct 2026

INC-MoE: Overlapping In-Network Collectives and Expert Computation for Expert-Parallel Mixture-of-Experts Inference

In expert-parallel mixture-of-experts (MoE) serving, dispatch moves tokens to the accelerators hosting their selected experts, and combine returns expert outputs to each token's owner. Both operations lie on the critical path and can dominate MoE layer latency in multi-node deployments. We present INC-MoE, an expert-pa...

John Stephan, Razvan-Stefan Puscasu, Yong Xu et al. · 0 citations
Preprint Nov 2026

Supercharging Olmo-core for Efficient and Scalable MoE Training

Mixture-of-Experts (MoE) models add parameters without proportionally increasing per-token compute. However, training cost also depends on bytes moved, memory held, and kernels launched, and many of those costs follow total parameter size rather than active compute. On infrastructure built for dense models, this mismat...

Tian-Hua Tao, Akshita Bhagia, Dirk Groeneveld et al. · 0 citations
Book Open access Oct 2026

CRISP: Co-designing Pruning and Scheduling for Efficient MoE Inference on Edge Servers

Mixture-of-Experts (MoE) inference on memory-constrained edge platforms is often dominated by expert-weight movement rather than computation. Although each token activates only a small subset of experts, routing is input-dependent, so all experts must remain accessible and memory movement dominates latency. Existing ap...

Cong-Wei Zhang, Youming Tao, Yi-Fei Zou et al. · 1 citation

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.