Low latency is critical to user experience. As NVIDIA Blackwell and Rubin GPUs advance multi-die scaling, data placement and cross-die communication become increasingly important. However, the latency implications and underlying costs of cross-die communication and die-aware execution remain largely unexplored. In this...
Rocky Wang, Joyce Zhou, Ke Jiang et al.· 0 citations
UlaCache is presented, a KV cache management system that provides unified logical addressing across GPU and CPU memory that reduces the overhead of KV replacement by parallelizing recency updates and eviction-candidate selection within a GPU memory pool shared across requests.
FlashBoot is presented, a hardware-friendly, framework-workflow co-designed weight-loading subsystem built on SGLang that accelerates single-node weight loading by up to 50x and concurrent rack-level weight loading by>270x and scales poorly to concurrent multi-node bring-up.
Issac Zhu, Hscos Zhang, Ke Jiang et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.