Jul 2026
SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling
SMetric addresses session-centric scheduling with differential scheduling based on two indicators derived from the request itself, the session turn and the local KV\$ hit: it schedules first-turn requests for load balance, and sticks follow-ups to the instance with the highest local hit for high KV\$ reuse.
Jiahao Wang, Kai-Zhan Lin, Kaixin Zhang et al.
· arXiv.org · 1 citation