Jun 2026· International Symposium on Computer Architecture· pp. 2142-2157· 0 citations· 67 references
Abstract
Efficient collective communication is crucial for distributed training. While recent topology-aware synthesis approaches attempt to optimize communication based on the network topology, they struggle with heterogeneous environments where links exhibit distinct bandwidths and asymmetric connections. The main inefficiency arises from single-round communication: toward the end of a communication phase, only a few remaining transfers occupy limited links, leaving most network channels idle. Furthermore, existing methods fail to handle the congestion that naturally occurs under overlapping execution, leading to significant bandwidth underutilization. To address these challenges, we propose PipeComm, a pipelineaware communication synthesis framework that maximizes bandwidth utilization by intelligently overlapping multiple data chunks. By explicitly modeling pipeline behavior, our method enables congestion-free scheduling across iterations and efficiently utilizes heterogeneous links. We develop an optimal synthesis method for constructing high-quality communication patterns and introduce a complementary incremental strategy that significantly improves scalability for large topologies. Experimental results show that the optimal strategy achieves over a 1.39× speedup compared to the state-of-the-art communication methods. Moreover, PipeComm supports diverse collective operations, demonstrating both efficiency and generality.
All-to-All collective communication is a key performance bottleneck for distributed machine learning (ML) and high-performance computing (HPC) workloads, where dense traffic increasingly stresses scale-up interconnects. While these ML and HPC workloads have driven unprecedented infrastructure demand, optical reconfigurable networks (ORNs) offer a promising path forward as they can reconfigure the network at runtime. By adapting the physical topology to the active workload, they improve communication cost and bandwidth utilization. However, optical reconfigurable networks introduce a fundamental trade-off for collective communication: each reconfiguration requires global synchronization, during which communication is suspended for at a non-negligible delay. Additionally, their benefit is critically contingent on whether the collective consists of structured phases that can be served by sparse and reusable topology states. In this paper, we revisit Bruck's All-to-All implementation and demonstrate the benefits of topology optimization in which both communication pattern and reconfiguration strategy are co-designed. We present ReTri, a bidirectional All-to-All schedule for ORNs based on the Trivance algorithm. ReTri uses balanced ternary block propagation to complete All-to-All in ⌈log3 n⌉ phases. The reconfiguration strategy induced by ReTri's pairwise bidirectional exchanges allows reconfiguration delays to be amortized across multiple phases. Preliminary simulations show that ReTri improves completion time by up to 10× over Pairwise All-to-All, even for millisecond-scale reconfiguration delays, and improves reconfigurable Bruck by up to 2.1×.
The rapid growth of data-intensive applications increases communication demands in many-core systems, where cache coherence, while essential for correct communication and data consistency, introduces substantial overhead due to frequent data sharing and coherence activities. As system scale and workload complexity grow, the resulting coherence traffic intensifies communication pressure, making the co-optimization of task mapping and routing essential for improving system performance. However, most existing approaches overlook cache coherence, leaving a substantial portion of coherence-induced communication unaccounted for and creating a mismatch between optimization objectives and actual communication patterns. Furthermore, by employing separate cost evaluators for mapping and routing, these approaches complicate objective coordination, may lead to conflicting decisions, and fail to capture the coherence-induced coupling between the two stages. To address these challenges, we propose CoCo, a coherence-aware co-optimization framework that jointly integrates task mapping and routing under a unified cost model for realistic scenarios. This unified model integrates communication cost, coherence overhead, and load imbalance into a single objective, enabling coherence-aware decision-making and effective trade-offs among optimization goals. Guided by this model, CoCo combines coherence-guided task mapping with reinforcement learning-based routing, where directional link weights are adjusted according to communication behavior to improve traffic distribution, enabling coherence-aware co-optimization for many-core systems. Experimental results show that CoCo reduces link utilization by 88.46%, packet delay by 17.40%, and execution time by 17.58% compared with existing approaches, highlighting the importance of cache coherence in co-optimization design.
Distributed LLM training requires GPUs to cooperate through collective communication primitives. While systems with proprietary high-speed interconnects have been extensively studied, the PCIe bandwidth in GPU clusters remains severely underutilized. Existing collective communication libraries treat the hierarchical PCIe tree as a flat GPU-to-GPU network and relegate the CPU to a passive data relay. Moreover, in systems equipped with dedicated interconnects, the PCIe path is largely overlooked. This paper introduces HyLink, a two-layer collective communication framework, including a PCIe CCL module and a Multipath Orchestrator module. PCIe CCL unlocks underutilized PCIe bandwidth by natively modeling the PCIe hierarchy and elevating the CPU to an active routing and computation node. It provides a CPU-driven Domain-Specific Language (DSL) for multi-stage data flows and an auto-pipelined transfer engine for host-side routing and SIMD-accelerated reduction. Multipath Orchestrator further enables hybrid-link acceleration by splitting each collective operation across the dedicated link and the PCIe bus. It employs a feedback-driven adaptive balancer to dynamically adjust the load ratio to accommodate runtime PCIe interference. Our experiments show that PCIe CCL achieves up to 2.28 × the bandwidth of NCCL on PCIe-only NVIDIA GPUs. HyLink improves bandwidth by up to 40% over HCCL on Ascend NPUs, and Multipath Orchestrator achieves 66% higher throughput than static splitting under dynamic PCIe interference.
Yue Liu, Menghao Zhang, Xuebin Song et al.· Asia-Pacific Workshop on Net...· 0 citations
Decentralized large language model (LLM) inference distributes transformer layers across heterogeneous nodes to enable resource-constrained participants to collaboratively serve large models. However, inter-node activation transfer constitutes a significant communication bottleneck, especially under low-bandwidth conditions prevalent in consumer-grade and edge networks. We propose BandwidthLLM, a communication-efficient framework that integrates three techniques: (1) a bandwidth-aware layer placement algorithm that minimizes boundary-level transfer cost according to link bandwidth and node reliability; (2) a lightweight activation compression scheme combining adaptive quantization with outlier-aware clipping and error feedback; and (3) a semantic preservation check that automatically falls back to higher precision when compressed activations deviate beyond a calibrated threshold. Experiments on Llama-2-7B, Mistral-7B, and Qwen2.5-7B across simulated bandwidth conditions (20–500 Mbps) demonstrate that BandwidthLLM reduces inter-node activation traffic by 62.7–63.4% relative to the BloomBee-like baseline while maintaining perplexity degradation of at most 0.31 points and task accuracy within 0.9 percentage points of uncompressed inference. Under 20 Mbps links, BandwidthLLM achieves 1.7× higher throughput and 41% lower latency.
Zhihao Wang, Shi Shu, Yulin Wang et al.· 2026 8th International Confe...· 1 citation
The Message Passing Interface (MPI) is a de-facto standard programming model for developing high-performance applications on large-scale supercomputers, where the performance of its collective communication operations over the underlying interconnection network is critical to the overall scalability of the applications. While the MPI defines the semantics of these operations, it leaves the algorithmic implementation to Open MPI, and selecting the algorithm typically relies on hard-coded decision logic. However, in real-world scenarios, this static approach often leads to sub-optimal performance, as it ignores the specific properties of the underlying system, particularly the complex and dynamic interconnection topologies. In this work, we address the challenge of selecting an efficient algorithm for a given collective, taking into account interconnection topologies. First, we demonstrate through a simulation study that interconnect topological features significantly influence the choice of the optimal collective algorithm for specific communication instances. Second, we propose a novel, machine learning-based, topology-aware framework for MPI collective algorithm selection. Third, we integrate this predictive framework directly into Open MPI, and show that our method is effective in selecting efficient collective algorithms for the Next Generation Tianhe Supercomputer, thereby unlocking the performance potential of the underlying interconnect.
Meng-Cheng Jia, Wei Zhang, Hui-Ze Zhang et al.· Fall Joint Computer Conferen...· 0 citations
Distributed large language model (LLM) systems increasingly rely on collective communication primitives such as AllReduce (AR), ReduceScatter (RS), AllGather (AG), and AlltoAll (A2A). In modern LLM training and serving clusters, heterogeneous GPU interconnects, multi-NIC networking, mixed parallelism strategies, low-latency inference requests, and high-throughput training pipelines have motivated increasingly diverse ways to plan, execute, and overlap collective communication. This paper presents a tutorial-style, collective-centric taxonomy for collective communication. We organize recent advances into three layers: communication planning, which generates topology-aware collective schedules; communication execution and adaptation, which maps these schedules onto GPU runtimes and hardware in real clusters; and computation-communication coordination, which turns collective optimization into end-to-end training and inference benefits. We further discuss open challenges and future opportunities for collective communication in distributed LLM systems.
Xuebin Song, Menghao Zhang, Yue Liu et al.· 0 citations