Long-context LLM inference is bottlenecked by GPU high-bandwidth memory (HBM), which the key–value (KV) cache rapidly exhausts. Existing designs compress, select, or offload KV entries, exposing trade-offs among HBM use, approximation error, and data movement. Our key insight is that locating the few tokens that domina...
Feng Yu, Hong-Shi Tan, Yao Chen et al.· Proceedings of the 2026 ACM...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.