Skip to content

Bridging LLM Serving and CXL-SSDs with Chunk-Aware KV Cache Management

Sep 2026 · 0 citations · 46 references
Computer Science

TL;DR

LM-CXD bridges the semantic gap between the serving engine and the device, and exposes NAND-to-DRAM progress to the serving engine, and uses device DRAM as a GPU-accessible buffer to hide NAND latency under limited device DRAM.

Abstract

NAND-backed storage offers the capacity needed to scale LLM prefix caching, but its block I/O path incurs CPU cache contention and host-DRAM staging in addition to NAND latency. Our characterization shows that these interface costs persist even with DRAM as the storage medium, motivating CXL-SSDs for byte-addressable access to NAND-backed capacity. Surprisingly, however, a stock CXL-SSD remains about 3$\times$ slower than local DRAM and no faster than an NVMe SSD, while generic prefetching provides little benefit. We present LM-CXD, a CXL-SSD specialized for LLM prefix caching. LM-CXD bridges the semantic gap between the serving engine, which knows which KV chunks will be consumed, and the device, which controls their placement and movement. It makes KV chunks device-visible I/O units, exposes NAND-to-DRAM progress to the serving engine, and uses device DRAM as a GPU-accessible buffer. LM-CXD further coordinates request scheduling with windowed prefetching and pipelines layerwise KV movement with GPU computation to hide NAND latency under limited device DRAM. Across five LLM models, LM-CXD reduces average TTFT over a stock CXL-SSD by up to 2.6$\times$ with compute asynchronous prefetching and 4.03$\times$ with layerwise prefetching, achieving TTFT within 1.5$\times$ of local DRAM on average.

View source

Similar papers

Preprint Oct 2026

A Flat KV Cache Storage Architecture for LLM Serving

KV cache reuse accelerates LLM serving, but existing systems manage host DRAM and NVMe SSD as a tiered hierarchy where DRAM acts as an intermediate cache. Production LLM workloads break both premises of this hierarchy. First, bimodal reuse defeats DRAM temporal filtering, since workloads are dominated either by one-sho...

Xin Huang, Teng Ma, Zhuo-Ran Li et al. · 0 citations
Preprint Oct 2026

SARIA: Persistence-Aware KV Caching with Coordinated DRAM--SSD Reload for LLM Agents

Multi-turn LLM agents usually reuse long context prefixes, making key value (KV) caching crucial for reducing prefill latency. SSDs provide a low-cost extension to limited DRAM capacity but introduce two key challenges. Eagerly persisting all KV cache blocks incurs substantial write traffic and rapidly consumes finite...

Hui Li, Qi-Hui Zhou, Xiao Yan et al. · 0 citations

TempoKV: Timely Staging of LLM KV Caches for Memory-Semantic Flash

TempoKV is presented, a timing-aware resource-commitment layer that separates early knowledge of reuse from the acquisition of staging resources when the runtime-estimated time until retrieval falls to the storage-estimated time needed to make KV resident and protected against eviction.

Jay H. Park, Hyungjun Kim, Dong Kim · 0 citations
#machine learning Preprint Oct 2026

Gandalf: Flash Shall Not Die! Increasing Flash Storage Lifetime for KV Cache Offloading in LLM Serving

KV caching reduces the serving cost of large language models (LLMs), but the growing KV cache footprint is increasingly exceeds GPU and host-memory capacity. Therefore, people use SSD to expand the KV cache storage capacity with high I/O bandwidth. However, in production LLM workload, such as multi-round conversation a...

Maharani A. P. Irawan, Kun-Tai Du, Jia-Yi Yao et al. · 0 citations
Preprint Oct 2026

Range: Decoupling KV-Cache Reuse Granularity from Storage Work

LLM serving reuses cached key-value (KV) state to skip prefill, and growing contexts push that state past GPU HBM and CPU DRAM onto NVMe SSDs, placing prefix restore on the critical path of time-to-first-token (TTFT). We find that on a well-configured host-issued path the SSD is not the bottleneck: restoring the same 6...

Ding-Sen Shi, Jia-Hao Zhang, Li-Qiang Lv et al. · 0 citations
Preprint Oct 2026

KVStream: Adaptive Parallel Storage Management for KV Caching in Large-Scale LLM Serving

Long-context LLM serving for agentic, coding, and document workloads reuses prefixes whose key-value (KV) cache outgrows GPU and host memory, putting storage I/O on the critical path of the inference step for retrieving prefixes. A single node-local NVMe absorbs all retrievals, while the parallel file system or cloud f...

Avinash Maurya, Jaiaid Mobin, Jie Ye et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.