DeferKV: Rethinking Eviction Timing for One-Shot KV Cache Compression
Long-context large language models (LLMs) have demonstrated strong capabilities across a wide range of tasks, but the growing KV cache introduces substantial memory and inference overhead. Existing one-shot KV cache compression methods typically commit to irreversible eviction immediately after prefill, before any sign...
Zhe Wang, Jia-Kai Li, Yu-Jia Sun et al.
· 0 citations