CXL Memory for LLM Inference Staging via an On-Device DMA Controller
LLM inference frequently exhausts GPU memory, forcing frameworks to stage data such as KV caches and intermediate tensors out of GPU HBM. Today this is done with GPUDirect Storage (GDS) over PCIe to NVMe SSDs, but NVMe bandwidth and latency remain a bottleneck for the fine-grained, high-frequency accesses these workloa...