Ke-Ming Hu

SparPR: Runtime KV-Cache Recovery for Resource-Constrained LLM Inference on Consumer Hardware

Running large language models (LLMs) on consumer hardware is bottlenecked by the key-value (KV) cache, which grows with every decoded token and quickly exhausts the memory and bandwidth of a single consumer GPU. We present SparPR (Sparse Partial Recomputation), a runtime KV-cache recovery mechanism that does not modify...