Rong Chen

Apto: Self-Adaptive CPU-GPU Hybrid MoE Inference on Shared PCs

CPU/GPU hybrid inference enables Mixture-of-Experts (MoE) models to run on memory-constrained PCs by keeping all expert weights in host memory and executing activated experts across both processors. Existing hybrid systems, however, assume exclusive resource access and determine an upfront CPU/GPU partition, so CPU con...