Mixture-of-Kittens: MoE Megakernel for NVL72s
AI accelerator systems are rapidly consolidating into scale-up architectures, where tens to thousands of GPUs communicate over high-bandwidth, single-hop fabrics. We find that existing Mixture-of-Experts (MoE) training systems, optimized for conventional scale-out networks, transfer poorly to this setting, often runnin...
Stuart H. Sul, Nash Brown, Henry Wildermuth et al.
· 1 citation
· ⚡1