T-CCL: Resource Efficient and Performant Collective Communication using Tensor Memory Accelerator
Large transformer-based models increasingly depend on multi-GPU execution, which requires frequent collective communication among GPUs. Existing communication libraries often rely on many GPU threads to achieve high bandwidth or low latency, resulting in a large streaming multiprocessor (SM)-side resource footprint. Th...
Keyvan Dadashzadeh, Yue-Hong Zhou, Min-Yu Cui et al.
· 0 citations