StepKV: Step-Aware KV Cache Compression for LLM Agents
Key-value (KV) caching is essential for efficient autoregressive large language model (LLM) inference, but the cache grows linearly with context length, increasing storage and decoding costs. KV cache compression mitigates this cost by retaining only a subset of cached tokens. This challenge is particularly important f...
Bo-Yu Feng, Jia-Hong Liu, Yi-Fan Li et al.
· 0 citations