Aug 2026· Proceedings of the ACM SIGCOMM 2026 Conference· pp. 1981-1987· 0 citations· 33 references
TL;DR
Dragonfly-Ultra is presented, a scalable, low-cost network architecture for high-performance AI clusters that can scale to over 260k GPUs with only 82% cost and 81% power consumption of a 3-layer Clos architecture and incorporates three key mechanisms to further improve network performance and optimize collective communication.
Abstract
Large-scale AI clusters impose higher requirements on network scalability, cost, and communication efficiency. The traditional Clos topology suffers from superlinear cost growth when scaling to over 100k GPUs, while the more cost-effective Dragonfly+ introduces "down-up" detours, deadlock risks, and complex routing design. This paper presents Dragonfly-Ultra, a scalable, low-cost network architecture for high-performance AI clusters. Dragonfly-Ultra can scale to over 260k GPUs with only 82% cost and 81% power consumption of a 3-layer Clos architecture. Dragonfly-Ultra optimizes inter-group connectivity to eliminate intra-group detours entirely. Beyond the topological benefits, Dragonfly-Ultra incorporates three key mechanisms to further improve network performance and optimize collective communication, including lightweight dual-waterline adaptive routing for fast congestion mitigation, virtual-link-based deadlock avoidance with lower hardware overhead, and uniform affinity-aware rank placement for balanced inter-group traffic across all phases. Simulation results on a 4k-node cluster show that, compared to Clos, Dragonfly-Ultra achieves up to 18.8% and 39.2% lower completion time for AllReduce and AlltoAll, respectively. Compared to Dragonfly+, the reductions are up to 27.9% and 62.1%, outperforming current mainstream topologies.
The rapid expansion of cloud computing, big data analytics, and artificial intelligence has positioned data centers as the backbone infrastructure of modern computing. Data center networks (DCNs) play a critical role in determining overall system performance and reliability. Existing DCN architectures face limitations, such as difficulties in balancing throughput and latency, insufficient fault-tolerance, and high expansion costs. To address these challenges, we propose ACDC (Augmented Cube-based Data Center), a novel server-centric DCN topology based on augmented cubes that achieves superior performance while maintaining cost-effectiveness through exclusive use of dual-port servers and low-port commodity switches. Firstly, we analyze the key features and properties of ACDC, with a focus on scalability and network diameter, establishing rigorous theoretical foundations for the proposed architecture. Secondly, we present comprehensive routing algorithms including ARouting for fault-free scenarios and AFR for fault-tolerant communication. Finally, extensive experimental evaluations demonstrate ACDC’s superior performance compared to state-of-the-art DCN architectures. Experimental results show that ACDC achieves a network diameter approximately 75% smaller than HSDC and 50% smaller than AQDN. Furthermore, ACDC maintains comparable throughput to the Fat-Tree under random traffic scenarios, while demonstrating substantial advantages under high-density all-to-all communication patterns, achieving at least 69.1% improvement in average throughput and at least 40.7% reduction in flow completion time compared to AQDN, DCell and FiConn. These confirm that ACDC strikes a good balance among performance, cost-efficiency, scalability, and fault-tolerance in contrast to the state-of-the-art DCN architectures.
Xuanli Liu, Weibei Fan, Zhenjiang Dong et al.· IEEE Transactions on Network...· 0 citations
Torus networks are deployed in production AI training clusters for their path diversity and low latency, but 2D Torus scales poorly: electrical packet switches compromise latency, and high-dimensional Torus introduces excessive routing complexity. We present STON (Scalable TOrus Network), a hierarchical architecture that treats a 2D Torus as a supernode and interconnects supernodes with a reconfigurable Optical Circuit Switch (OCS) for AlltoAll-dominated large-scale training networks. STON comprises three coordinated modules: (1) fragmentaware task placement, which minimizes inter-supernode traffic by reducing job fragmentation; (2) non-disruptive logical topology mapping, governed by two principles that prevent OCS reconfiguration from disrupting running tasks or partitioning multisupernode jobs; and (3) compute-phase traffic forwarding, which ensures reachability when direct OCS circuits are unavailable. STON reduces average FCT by 42.2%-61.1% across synthetic workloads and by 52.6% on a one-day Kalos production trace (under an AlltoAll traffic model for all jobs), with 95th-percentile tail latency reduced by up to 74.5%, versus a static direct-connect baseline using the same OCS hardware.
Qinwei Yang, Peirui Cao, Ruyi Zhang et al.· Fall Joint Computer Conferen...· 0 citations
All-to-All collective communication is a key performance bottleneck for distributed machine learning (ML) and high-performance computing (HPC) workloads, where dense traffic increasingly stresses scale-up interconnects. While these ML and HPC workloads have driven unprecedented infrastructure demand, optical reconfigurable networks (ORNs) offer a promising path forward as they can reconfigure the network at runtime. By adapting the physical topology to the active workload, they improve communication cost and bandwidth utilization. However, optical reconfigurable networks introduce a fundamental trade-off for collective communication: each reconfiguration requires global synchronization, during which communication is suspended for at a non-negligible delay. Additionally, their benefit is critically contingent on whether the collective consists of structured phases that can be served by sparse and reusable topology states. In this paper, we revisit Bruck's All-to-All implementation and demonstrate the benefits of topology optimization in which both communication pattern and reconfiguration strategy are co-designed. We present ReTri, a bidirectional All-to-All schedule for ORNs based on the Trivance algorithm. ReTri uses balanced ternary block propagation to complete All-to-All in ⌈log3 n⌉ phases. The reconfiguration strategy induced by ReTri's pairwise bidirectional exchanges allows reconfiguration delays to be amortized across multiple phases. Preliminary simulations show that ReTri improves completion time by up to 10× over Pairwise All-to-All, even for millisecond-scale reconfiguration delays, and improves reconfigurable Bruck by up to 2.1×.
This work proposes a novel topology named the Balanced Sparse Tree (BST), which is a topology characterized by symmetric design and sparse connections, motivated by hypergraph theory and Steiner Systems, and demonstrates the superiority of BST over the state-of-the-art in network scale, latency, bandwidth, and cost.
Shaoteng Liu, Dejun Kong, Huitian Wang et al.· 0 citations
Mixture-of-Experts (MoE) models outperform traditional dense models through sparse expert activation, where each token is dynamically routed to a small subset of experts. Across many tokens, these sparse Dispatch operations induce all-to-all traffic, making communication a major bottleneck for both training and inference: unicast replication wastes bandwidth, while table-driven multicast struggles with receiver-set churn and incast. In this paper, we propose SprayCast, a congestion-adaptive native RDMA multicast scheme for dynamic sparse token Dispatch. To avoid maintaining multicast forwarding tables in switches, SprayCast encodes each packet’s destination node set in its packet header using hierarchical bitmaps, enabling table-free in-network replication. It uses in-band network telemetry (INT) feedback to steer replication away from congested multicast branches and range-based negative acknowledgments (NACKs) for localized loss recovery, saving bandwidth and reducing tail latency in dynamic all-to-all communication. In htsim simulations on a 128-server fat-tree, SprayCast achieves better scalability as top-K dispatch fanout increases and reduces P99 dispatch tail latency by up to 6 × at K = 8 compared with representative baselines.
Yingying Zeng, Ning Xu, Xiaobin Tan et al.· Asia-Pacific Workshop on Net...· 0 citations
With the rapid growth of large language model training and generative artificial intelligence services, data center networks face severe micro-burst traffic and high concurrency. Traditional hash-based flow-level load balancing cannot sense link states, leading to hash collisions, hotspot congestion, and tail latency in multipath Clos networks. Existing packet-level schemes are constrained by stale state information, high hardware complexity, and poor adaptation to heterogeneous links. To address these issues, this paper proposes probabilistic state-proportional (PSP) dispatching, a packet-level load balancing algorithm. Using a Band-based discrete state representation, PSP replaces global sorting with local probability mapping, reducing hardware complexity while suppressing herding and oscillations caused by stale states. Experiments on a cycle-accurate simulator show that PSP is robust across port scales, bandwidth-limited paths, and fixed-flow interference. It outperforms join-the-shortest-queue (JSQ) scheduling and Random in loss rate, 99th-percentile buffer occupancy, and scalability, while remaining competitive with Top-k at lower hardware cost. PSP provides an effective balance among performance, stability, and overhead for artificial intelligence data centers.
Jiaqi Liu, Chunyang Zhang, Heng Pan et al.· 0 citations