Preprint
Aug 2026
Compute-Optimal Is Not Cluster-Optimal: Systems-Aware Scaling for Sparse Mixture-of-Experts
MOSAIC is developed, which formulates model architecture and systems co-design as an optimization problem that couples a predictive scaling law with a calibrated performance model that estimates Model FLOPs Utilization (MFU), communication cost, memory footprint, and the best parallel layout.
Soumajyoti Sarkar, Yu-Xin Tang, Sheng Zha
· 0 citations