LiquidCP: Decoupling Attention Tasks from Communication and Kernel Boundaries in Context Parallelism
Context parallelism (CP) partitions long sequences across GPUs for LLM training and inference. For variable-length training and full prefill, attention-balanced placement can leave token loads uneven, while ring execution repeats overhead at each computation and communication step. In stage-serialized decode CP (DCP),...