The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction
Edge0 is presented, a streaming MoE inference engine that closes the gap with a prerouter: a per-layer head predicts the next layer's routing one token ahead, and the prediction is consumed as the routing itself, so the staged expert set equals the routed set and nothing is dropped.
Yu Lin, Yi-Ming Wang, Run-Yuan Cai et al.
· 0 citations