Aug 2026· Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication· 0 citations· 18 references
Computer Science
TL;DR
Giganetes is a GPU-centric architecture for stateless LLM serving that externalizes KV caches to a disaggregated remote memory pool via GPUDirect RDMA, eliminating session affinity constraints and enabling near-linear horizontal scaling in a Kubernetes-native deployment.
Abstract
Giganetes is a GPU-centric architecture for stateless LLM serving that externalizes KV caches to a disaggregated remote memory pool via GPUDirect RDMA. By treating remote memory as a GPU-addressable tier via GPUDirect RDMA, Giganetes eliminates session affinity constraints: any GPU can serve any request, enabling near-linear horizontal scaling in a Kubernetes-native deployment. A Scatter/Gather I/O interface bypasses the CPU and host memory entirely, achieving 52.4 GB/s application-level read throughput on our 4×200 Gbps RDMA testbed. A session-level metadata abstraction and proactive readahead mechanism reduce GPU bubbles by overlapping remote KV fetches with prefill computation and scheduling slack. On a 4-node H800 cluster, Giganetes delivers 33% higher throughput (QPS 2.4 vs. 1.8) and 1.75× lower P95 TPOT than PD-disaggregation with sticky sessions, with the gain driven by scheduling flexibility rather than faster transport alone.
This study describes GPU-fabric contention in LLM workloads and presents GPUWeaver, a context-aware communication scheduler that monitors application progress and fabric state, identifies contention, and dynamically regulates communication injection according to application performance goals.
Dan-Yang Chen, Yu-Feng Gu, Yi-Bo Huang et al.· Proceedings of the 17th ACM...· 2 citations
GPUs are expensive resources but often underutilized. This inefficiency is exacerbated by the increasing diversity of AI workloads, where small or specialized models often fail to saturate high-capacity devices, necessitating GPU sharing. Existing sharing approaches force a hard tradeoff. They either partition hardware...
Yi-Cheng Liu, Yi-Fan Qiao, Tian Xia et al.· Proceedings of the 32nd ACM...· 0 citations
InplaceKVCache is proposed, the first KVCache abstraction whose format fixes each byte's physical residency at write time, so that the CPU--GPU load balance can be adjusted without moving data after placement, turning load balancing into pure scheduling.
This work presents AsymFlow, a prefill-decode disaggregated serving system that runs prefill on the GPU and decode on the CPU, and implements on SGLang and evaluated on a CPU-GPU platform.
Jun-Wen Zhang, Wei-Ling Yang, Jian-Bin Fang et al.· International Conference on...· 1 citation
Ghost is an OS-level GPU virtualization layer integrated directly into the open-source GPU driver, using a GPU container abstraction with cgroup -like APIs for compute and memory control and privileged hardware-level scheduling and preemption for dynamic compute resource management.
DynamoServe is presented, a multi-tenant LLM serving framework that addresses challenges through three key innovations: leveraging stranded GPU memory to offload model weights and KV caches, mitigating resource fragmentation in multi-workload environments, and improving memory locality through coordinated data placemen...
Diman Zad Tootaghaj, Khaled Diab, Bob Lantz et al.· Conference on Applications,...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
