Open access
Aug 2026
SwiftQK: Fast and Communication-Efficient Tensor Parallelism for Query-Key Normalization
SwiftQK is presented, a multi-GPU RMSNorm kernel that exchanges only scalar normalization statistics and overlaps the remaining Peer-to-Peer reduction with independent element-wise computation in a deadlock-safe persistent kernel.
Gyudong Kim, Wonjun Han, Young Geun Kim
· IEEE computer architecture l... · 0 citations