CompKV: Compensation-Aware KV Selection for Long-Context LLM Inference
CompKV is introduced, the first compensation-aware sparse attention framework that divides tokens into blocks and explicitly optimizes selection for the downstream compensation mechanism, and shows that the residual left by block-level mean compensation is governed by both block attention mass and within-block logit va...
Zheng-Hong Huang, Rui-Zhe Yao, Dan-Yi Liu et al.
· 0 citations