HyLink: Harnessing PCIe and Dedicated Interconnects for Efficient Collective Communication

Abstract

Distributed LLM training requires GPUs to cooperate through collective communication primitives. While systems with proprietary high-speed interconnects have been extensively studied, the PCIe bandwidth in GPU clusters remains severely underutilized. Existing collective communication libraries treat the hierarchical PCIe tree as a flat GPU-to-GPU network and relegate the CPU to a passive data relay. Moreover, in systems equipped with dedicated interconnects, the PCIe path is largely overlooked. This paper introduces HyLink, a two-layer collective communication framework, including a PCIe CCL module and a Multipath Orchestrator module. PCIe CCL unlocks underutilized PCIe bandwidth by natively modeling the PCIe hierarchy and elevating the CPU to an active routing and computation node. It provides a CPU-driven Domain-Specific Language (DSL) for multi-stage data flows and an auto-pipelined transfer engine for host-side routing and SIMD-accelerated reduction. Multipath Orchestrator further enables hybrid-link acceleration by splitting each collective operation across the dedicated link and the PCIe bus. It employs a feedback-driven adaptive balancer to dynamically adjust the load ratio to accommodate runtime PCIe interference. Our experiments show that PCIe CCL achieves up to 2.28 × the bandwidth of NCCL on PCIe-only NVIDIA GPUs. HyLink improves bandwidth by up to 40% over HCCL on Ascend NPUs, and Multipath Orchestrator achieves 66% higher throughput than static splitting under dynamic PCIe interference.