Democratizing MoE inference on commodity GPUs with CoMoE
Deploying Mixture-of-Experts (MoE) models relies heavily on Expert Parallelism, which generates intense inter-GPU communication. Consequently, state-of-the-art inference systems require high-bandwidth, P2P interconnects (e.g., NVLink) in datacenter GPUs to handle massive token routing, making deployment prohibitively e...
Ru-Wen Fan, Y. Zu, Min-Hui Xie et al.
· 0 citations