GrowPage: On-Demand KV Budgeting for Efficient LLM Reasoning Serving
Long-output reasoning has made the key--value (KV) cache a critical memory bottleneck for efficient LLM serving. Existing KV compression methods usually rely on a predefined per-request budget and adjust only which KV states are retained, leaving the total capacity fixed throughout decoding. However, reasoning workload...
Qian-Kun Ma, Yan-Jiang Zhou, Zi-Nan Xiong et al.
· 0 citations