ResidualKV: Residual-Based KV Cache Compression for Efficient Long-Context Inference
ResidualKV is proposed, which factorizes the KV cache into a sparse set of globally retrieved references and compact, quantized residual codes for the remaining tokens, and shows that global cross-token redundancy supports accurate, memory-efficient, and computation-efficient long-context inference.
Jitai Hao, Qiang Huang, Yaowei Wang et al.
· 10 citations
· ⚡1