Preprint
Aug 2026
Heterogeneous LLM Serving with General-Purpose Processing-Near-Memory for Retrieval-Based Sparse Attention
A heterogeneous decode-phase serving system that relocates the KV cache out of GPU memory, motivated by the retrieval-based sparse attention that recent frontier LLMs adopt to serve million-token contexts and proposes KARAT, a general-purpose PNM design that is the design point meeting all four requirements.
Hyungkyu Ham, Junhyeong Bae, Seungheon Lee et al.
· 1 citation