Argus: Hardware-Software Co-Scaling of Tensor Parallelism for Efficient LLM Training
Abstract
Training LLM requires hybrid parallelism across thousands of GPUs, yet tensor parallelism (TP)—partitioning large weight tensors—remains limited to small degrees in practice due to hardware and software scalability constraints. This limited TP degree forces deeper pipeline parallelism or larger data parallelism, each incurring additional inefficiencies. It also increases memory bandwidth demand, placing an unnecessary burden on already capacity-constrained training systems. Driven by these limitations, we advocate scalable TP, where the TP degree scales with the model hidden size. We realize this with Argus, a communication- and memory-efficient system co-designed across hardware and software. On the hardware front, Argus leverages a switchless chiplet architecture for low link latency and high scalability. We further introduce a split-memory design that consistently prevents memory access from becoming a bottleneck at scale under physical and cost constraints. On the software front, we develop a TP solution with low communication complexity, and adopt an interleaved transfer pattern tailored for connectivity-limited switchless networks. We further remove the TP-degree constraint imposed by attention head count by eliminating partial-sum transfer overhead. Collectively, these optimizations enable coordinated scaling: computation, communication, and memory overheads remain asymptotically constant as the TP degree increases. This property is validated by both theoretical analysis and experiments, with Argus delivering 3.10×/2.02× speedup/energy saving over typical TP baselines. Scalable TP further improves hybrid-parallel training efficiency by better balancing different forms of parallelism, yielding an estimated 65% MFU for Llama3.1-405B.