HyCo: Hybrid Drafting for Cost-Efficient Mobile-Cloud Speculative Decoding
Device-cloud speculative decoding reduces server computation, but growing a deep draft tree with a small language model (SLM) requires repeated passes on a mobile neural processing unit (NPU). We present HyCo, which assigns depth to the server model's native multi-token prediction (MTP) head, executed on the device, an...
Hanbin Bae, Heejin Choi, Joohyung Lee et al.
· 0 citations