JetFire: Approximate-then-Correct Attention on Hybrid Memory for Long-Context LLM Inference
Abstract
Long-context LLM inference is bottlenecked by GPU high-bandwidth memory (HBM), which the key–value (KV) cache rapidly exhausts. Existing designs compress, select, or offload KV entries, exposing trade-offs among HBM use, approximation error, and data movement. Our key insight is that locating the few tokens that dominate attention needs far less precision than computing the output over them: a heavily quantized cache can no longer reproduce the output, yet it still ranks tokens reliably. JetFire exploits this asymmetry with an approximate-then-correct (ATC) attention backend over hybrid memory. A dense low-precision pipeline scans a compact replica of the whole sequence in GPU HBM, fitting long contexts without dropping tokens. A sparse high-precision pipeline on the CPU corrects the dominant tokens from the full-precision replica in host DRAM. The correction overlaps the GPU value phase when transfer and computation fit within it. JetFire remains near-lossless relative to full attention, and JetFire-K3V2 achieves 1.3–3.0 × speedup in end-to-end execution time over TurboQuant within vLLM at the largest common contexts. Against offloading-based systems, it achieves 1.4–2.7 × speedup over NEO and 13–180 × over InfiniGen. JetFire is open-sourced at https://github.com/Xtra-Computing/JetFire.