Context parallelism (CP) partitions long sequences across GPUs for LLM training and inference. For variable-length training and full prefill, attention-balanced placement can leave token loads uneven, while ring execution repeats overhead at each computation and communication step. In stage-serialized decode CP (DCP),...
Hong-Yu Chen, Yi-Kang Ruan, Yu-Hai Long et al.· 0 citations
This work proposes FIBER, a new architecture that extends the GPU SIMT (single instruction, multiple thread) model, and extends the ISA, microarchitecture, and compiler to realize shared-register addressing, conflict-free operand delivery, and fiber-based program mapping.
Zi-Han Liu, Jingwen Leng, Yang-Jie Zhou et al.· 0 citations
This work proposes GPU-Tile-Sim, a tile-centric GPU simulation framework for LLM hardware-software co-design that represents kernel execution as a warp-level tile graph whose nodes capture tile-level operations and whose edges encode data and ordering constraints.