This work presents Coda, a coding-agent serving system that realizes admission flexibility through a readiness-informed admission layer incorporating two mechanisms: Tiered-Aging state admission exploits bounded flexibility in admission order to improve admission efficiency and preserve request progress.
You-He Jiang, Fang-Cheng Fu, Bin-Hang Yuan et al.· 0 citations
Long-context large language model inference is bottlenecked by KV caches that grow linearly with sequence length. This burden is especially severe for long, reusable context prefixes, whose cache must serve many downstream queries. Reconstruction-based methods such as Attention Matching achieve strong downstream task p...
Zhe-Yu Shen, Guan-Hua Wang, De-Zhan Tu et al.· 1 citation
This work presents Crossflow, which makes this boundary elastic without changing node roles, and improves token throughput by 16.2-17.4% on geometric mean over static P/D, and by up to 43.4% at high load, while reducing mean TTFT at every evaluated point.
Yi Xu, Ehsan K. Ardestani, Wen-Yin Fu et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.