FlashOffloading: Low-Latency KV Cache Offloading for Long-Context LLM Serving
FlashOffloading is presented, a host-DRAM-centric inference architecture that fully offloads the KV cache in both the prefill and decode phases, optimizing KV offloading latency across the entire pipeline and delivering throughput gains under a time-per-output-token (TPOT) service-level objective (SLO).
Xuan Wang, Zhuan-Hao Wu, Ting-Zhu Fang et al.
· 0 citations