AI Networking Cookbook: Practical recipes for AI-assisted network automation and development
TL;DR
It is argued that a KV cache server — disaggregated, CXL-attached memory device(s) with optional near-memory compute—is the correct infrastructure primitive for the next generation of AI data centers.
This work presents InferScale, a GPU-native LLM memory system that replaces repeated prompt prefilling with reusable KV state, and encodes each memory fact together with a small window of preceding conversation context while caching only the target fact's KV.
OasisKV is presented, a memory-centric LLM inference system design that alleviates HBM capacity pressure by decoupling full KV-cache storage from HBM during LLM decoding and observes that future important tokens can be predicted accurately in advance using lookahead tokens drafted by speculative decoding (SD).
DualDecoder is presented, a lightweight serving system for long-context LLM inference that enables efficient sparse KV cache retrieval from host memory that leverages a novel dual-token decoding pipeline that accurately identifies critical KV entries with negligible computational overhead.
FLINT is proposed, a workload-driven HBF substrate for capacity-scalable LLM inference that integrates HBF as a memory-capacity tier alongside HBM while addressing three adoption challenges.
This model reveals one key opportunity: dividing a restore request proportionally between the storage path and the GPU can improve inference performance while still meeting SLOs, and reduces the KV-cache storage stack to a performance model based on per-tier capacity, per-tier and interconnect bandwidth, and GPU arithm...
This paper presents HyMCache, a CXL memory rack for multi-turn LLM serving using cost-efficient CXL-hybrid memory, which combines a small amount of in-device DRAM with large SSD-backed capacity behind a CXL interface to efficiently support TB-scale SSD-backed KV reuse.
We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.