Online LLM serving is increasingly constrained by GPU memory capacity during decode. In length-skewed workloads, a small fraction of long requests keeps large key-value (KV) caches resident across many decode steps, exhausting GPU high-bandwidth memory (HBM), throttling admission, and reducing goodput for the short-req...
Dong Chen, Yu-Kun Zhou, Sheng Yao et al.· Proceedings of the 2026 ACM...· 0 citations
LLM-powered multi-agent systems are increasingly deployed in production, yet diagnosing their failures remains a major operational challenge. Unlike failures in cloud architectures, which can often be diagnosed using structured diagnostic information, multi-agent system failures are embedded in unstructured natural-lan...
Yong-Xin Zhao, Sheng-Lin Zhang, Wen-Wei Gu et al.· Proceedings of the 2026 ACM...· 0 citations
LotteRig is presented, a proactive availability enhancement framework that performs risk-aware GPU selection and cost-aware error mitigation, and achieves 9.9% fewer software interruptions and 12.1% higher hardware availability across diverse scenarios, substantially improving the odds of ‘winning’ the GPU resilience l...
Xin-Kai Wang, Yun-Wei Li, Chao Li et al.· ACM Transactions on Architec...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.