Compute-Communication Overlap Is Not Free: A Cross-Layer Characterization in GPU LLM Workloads

Abstract

Distributed large language model (LLM) systems rely on compute-communication (CC) overlap to hide communication overhead. However, scheduling compute and communication concurrently and maximizing overlap opportunities do not guarantee effective overlap. Resource contention can slow concurrent operations, while communication that exceeds the available computation time remains exposed on the critical path. We study these limitations using end-to-end LLaMA 3 and Mixtral workloads on single-node H100 and H200 systems and multi-node H100 systems, complemented by targeted microbenchmarks. On a single node, concurrent execution slows kernels by 20.4–55.8% relative to sequential execution, substantially reducing the expected benefit of overlap. In multi-node workloads, kernel slowdown can be modest, but execution is still up to 2.00× slower than an ideal overlap because communication often outlasts computation.Our microbenchmarks trace these inefficiencies to interactions across the hardware, runtime, and kernel layers. The main contributors are contention for streaming multiprocessors and memory bandwidth, limitations in launch ordering and stream priorities, amplification of stragglers across ranks, and communication kernels that rely heavily on polling. These results show that future systems must optimize CC overlap efficiency, not only overlap opportunity, through coordinated runtime, kernel, and hardware support.