Ze-Lei Shao

Osprey: Target-agnostic Pre-training Makes Stronger Drafters in Speculative Decoding

Speculative decoding is critical for accelerating LLM inference. However, the speedup is fragile: drafters are typically trained against a narrow distribution for a single target model, and their acceptance rate collapses under workload shifts. This is a striking inversion of modern LLM development, where target models...