Xiao-Peng Deng

HiNa-MoE: High-Performance, Non-Intrusive MoE Inference on CPUs with Matrix Engines

Mixture-of-Experts (MoE) inference is increasingly deployed in local and on-premise environments, where expert parameters often exceed GPU memory capacity. In latency-sensitive, low-concurrency settings, repeatedly staging routed-expert weights from CPU memory to the GPU can be prohibitive, leaving routed-expert feed-f...