Shuaikang Hou

ATAC: Anchor-tail aware context parallelism for LLM training

This work proposes ATAC, an anchor-tail aware framework for jointly optimizing packed-document construction and context-parallel sharding in large language model training and demonstrates that input-structure-aware co-design of packing and context parallelism is an effective approach to improving calibrated pipeline-le...