OptiFlow: Towards LLM-Driven Optimization of Collective Communication Algorithms
TL;DR
OptiFlow is presented, among the first LLM-driven frameworks for automated design of high-performance collective communication algorithms, with key insight is a two-layer decomposition: the LLM generates compact data-movement intent expressed in a domain-specific language, while deterministic scheduling algorithms compile these programs into executable schedules.
Abstract
Training trillion-parameter models has made collective communication a dominant bottleneck in GPU clusters. Existing approaches face a fundamental tension: optimization-based synthesis techniques suffer from combinatorial complexity that can require minutes to hours, while analytic heuristics rely on rigid assumptions that often fail under production variability. To the best of our knowledge, we present OptiFlow, among the first LLM-driven frameworks for automated design of high-performance collective communication algorithms. Our key insight is a two-layer decomposition: the LLM generates compact data-movement intent expressed in a domain-specific language, while deterministic scheduling algorithms compile these programs into executable schedules. We further employ an iterative pipeline that uses real-hardware feedback to refine LLM-generated proposals, without updating model parameters or relying on manually designed search heuristics. Evaluated on a 32-GPU NVIDIA A100 cluster, OptiFlow discovers All-Gather schedules that outperform NCCL by up to 3.75 × , while it also surpasses TACCL and TE-CCL with gains up to 5.12 × and 3.28 × , and shows better optimization-time scalability.