KV-Repurpose: A Testbed for Evaluating Non-Prefix KV-Cache Reuse
KV-cache reuse in large language model serving is commonly limited to exact prompt prefixes, preventing reuse when the same retrieved document appears after different system prompts, conversation histories, or dynamic metadata. We present KV-Repurpose, an open-source, vLLM-based system for developing and evaluating seg...
Taiga Ikeda, Daisuke Miyashita, Jun Deguchi
· 1 citation