Enabling Efficient All-Dimension Top-k Sparsification for High-Performance Distributed DNN Training Systems
ADTopk is proposed, an all-dimension top-k sparsification scheme that selects the k largest elements across all dimensions of the per-layer gradient tensor, ensuring that every dimension contributes at least one element and eliminating dimension missing.