SprayCast: Congestion-Adaptive Native Multicast for Dynamic Sparse All-to-All Communication

Abstract

Mixture-of-Experts (MoE) models outperform traditional dense models through sparse expert activation, where each token is dynamically routed to a small subset of experts. Across many tokens, these sparse Dispatch operations induce all-to-all traffic, making communication a major bottleneck for both training and inference: unicast replication wastes bandwidth, while table-driven multicast struggles with receiver-set churn and incast. In this paper, we propose SprayCast, a congestion-adaptive native RDMA multicast scheme for dynamic sparse token Dispatch. To avoid maintaining multicast forwarding tables in switches, SprayCast encodes each packet’s destination node set in its packet header using hierarchical bitmaps, enabling table-free in-network replication. It uses in-band network telemetry (INT) feedback to steer replication away from congested multicast branches and range-based negative acknowledgments (NACKs) for localized loss recovery, saving bandwidth and reducing tail latency in dynamic all-to-all communication. In htsim simulations on a 128-server fat-tree, SprayCast achieves better scalability as top-K dispatch fanout increases and reduces P99 dispatch tail latency by up to 6 × at K = 8 compared with representative baselines.