Odin: Rethinking Congestion Control under All-to-All Traffic
TL;DR
Odin is proposed to enhance CC under A2A traffic, which achieves low TBO and high throughput under A2A traffic, accelerating the training process and keeping strong performance under non-A2A traffic.
Abstract
All-to-All (A2A) communication dominates the training and inference time of large language models, especially under the Mixture-of-Experts (MoE) architecture where expert dispatch induces dense concurrent flows. We observe that A2A traffic consistently builds up large total buffer occupancy (TBO) at switches. Analyzing the accompanying counterintuitive phenomena, we reveal that the A2A traffic saturates ingress bandwidth, interferes with rate adjustment of congestion control (CC) through sparse congestion signals and rate inflation, and causes ingress-egress bandwidth mismatch. Unaware of these issues, existing CC algorithms would be easy to trigger PFC with large TBO, suffer from ineffectual congestion response, and lead to throughput loss. Based on these insights, we propose Odin to enhance CC under A2A traffic. Odin suppresses large TBO based on the switch-level TBO signals, proactively allocate ingress bandwidth to eliminate rate inflation, and make bandwidth compensation for ingress-egress matching. Experiments and simulations show that Odin achieves low TBO and high throughput under A2A traffic, accelerating the training process. Meanwhile, it keeps strong performance under non-A2A traffic.