ARC-KV: Amortizing Anchor Search for Reconstruction-Based KV Cache Compaction
Long-context large language model inference is bottlenecked by KV caches that grow linearly with sequence length. This burden is especially severe for long, reusable context prefixes, whose cache must serve many downstream queries. Reconstruction-based methods such as Attention Matching achieve strong downstream task p...
Zhe-Yu Shen, Guan-Hua Wang, De-Zhan Tu et al.
· 0 citations