Bridging LLM Serving and CXL-SSDs with Chunk-Aware KV Cache Management
NAND-backed storage offers the capacity needed to scale LLM prefix caching, but its block I/O path incurs CPU cache contention and host-DRAM staging in addition to NAND latency. Our characterization shows that these interface costs persist even with DRAM as the storage medium, motivating CXL-SSDs for byte-addressable a...
Hyunsun Chung, Taewan Noh, Minji Kim et al.
· 0 citations