Verify Before You Fetch: Reliable KV Cache Compression for LLM Inference
VerKV enables attention with bounded approximation error while minimizing CPU-to-GPU transfers, and it is shown that PoissonComp outperforms state-of-the-art approaches to KV cache compression.
Devanshu Jha, G. Kumar, Agniva Chowdhury et al.
· 0 citations