Skip to content
Book Open access

Argus: Hardware-Software Co-Scaling of Tensor Parallelism for Efficient LLM Training

2027 · Proceedings of the 32nd ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1 · 0 citations · 17 references

Abstract

Training LLM requires hybrid parallelism across thousands of GPUs, yet tensor parallelism (TP)—partitioning large weight tensors—remains limited to small degrees in practice due to hardware and software scalability constraints. This limited TP degree forces deeper pipeline parallelism or larger data parallelism, each incurring additional inefficiencies. It also increases memory bandwidth demand, placing an unnecessary burden on already capacity-constrained training systems. Driven by these limitations, we advocate scalable TP, where the TP degree scales with the model hidden size. We realize this with Argus, a communication- and memory-efficient system co-designed across hardware and software. On the hardware front, Argus leverages a switchless chiplet architecture for low link latency and high scalability. We further introduce a split-memory design that consistently prevents memory access from becoming a bottleneck at scale under physical and cost constraints. On the software front, we develop a TP solution with low communication complexity, and adopt an interleaved transfer pattern tailored for connectivity-limited switchless networks. We further remove the TP-degree constraint imposed by attention head count by eliminating partial-sum transfer overhead. Collectively, these optimizations enable coordinated scaling: computation, communication, and memory overheads remain asymptotically constant as the TP degree increases. This property is validated by both theoretical analysis and experiments, with Argus delivering 3.10×/2.02× speedup/energy saving over typical TP baselines. Scalable TP further improves hybrid-parallel training efficiency by better balancing different forms of parallelism, yielding an estimated 65% MFU for Llama3.1-405B.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.