Plug-and-Play KV Cache Reuse for Low-Latency Retrieval-Augmented Generation
TL;DR
This work studies KV cache reuse for RAG under a strictly plug-and-play regime, where the base language model is left unchanged and no retraining, prompt redefinition, or selective recomputation is performed, and proposes a fine-tuning-free approach that reuses KV states across queries through an anchor-block and position- adjustment strategy.
Abstract
Retrieval-augmented generation (RAG) systems often suffer from high latency during the prefilling stage, when large language models compute key–value (KV) caches for long prompts. Although caching shared documents can reduce redundant computation, directly reusing precomputed KV blocks typically causes substantial accuracy loss.In this work, we study KV cache reuse for RAG under a strictly plug-and-play regime, where the base language model is left unchanged and no retraining, prompt redefinition, or selective recomputation is performed. Within this setting, we propose a fine-tuning-free approach that reuses KV states across queries through an anchor-block and position-adjustment strategy. Our design preserves alignment with the model’s attention mechanism while enabling efficient recombination of cached blocks.Experiments on multiple QA benchmarks demonstrate up to a 5.9× reduction in time-to-first-token (TTFT) with only moderate accuracy degradation, demonstrating a practical efficiency–accuracy trade-off and establishing KV reuse as a promising path for real-time RAG applications where system simplicity and inference speed are the primary priorities.