Deadline-Aware Adaptive Prefill Chunking for Efficient Large Language Model Serving
Continuous batching improves large language model (LLM) serving throughput, but long prompt prefills can delay decode iterations and violate inter-token latency objectives. Chunked prefill mitigates this interference, yet its chunk size is normally fixed: small chunks protect decode latency but repeatedly pay launch ov...
Si-Yu Song, Qi-Wei Bai, Jin-Bo Hao et al.
· 1 citation