EfficientAgent: What Makes KV Cache Offloading Work for Concurrent Agents?
LLM agents resend their whole conversation on every turn, and most of it was already processed on the previous turn. Serving systems avoid recomputing it by caching its key-value (KV) state and, when GPU memory runs out, by offloading that state to host memory. For agents, offloading gives inconsistent results: on the...
Kun-Ming Shao, Jie-Run Chen, Jiang-Nan Yu et al.
· 0 citations