StreamDQ is proposed, a lightweight architectural enhancement that enables on-the-fly dequantization in the memory subsystem for high-throughput, large-batch LLM inference and reduces latency and improves decode throughput for end-to-end LLM inference.
Abstract
As large language models (LLMs) scale, their memory and computation demands have grown substantially, making weight-only quantization a widely adopted technique for reducing model size with minimal accuracy loss. However, on current GPUs, CUDA-core-based dequantization introduces substantial instruction overhead, on-chip traffic, and pipeline stalls, making it a major bottleneck for high-throughput, cloud-scale LLM serving. To address these limitations, we propose StreamDQ, a lightweight architectural enhancement that enables on-the-fly dequantization in the memory subsystem for high-throughput, large-batch LLM inference. StreamDQ integrates compact DeQuantization Blocks (DQBs) into the base die of high-bandwidth memory (HBM) and performs inline dequantization on standard memory loads. A lightweight sideband tag on each memory read request selects the dequantization mode while preserving conventional load semantics. By relocating dequantization to the memory side, StreamDQ eliminates GPU-side CUDA-core-based dequantization, thereby reducing on-chip traffic on the GPU and avoiding extra HBM write-back and reload of dequantized weights at large batch sizes. Our evaluation shows that StreamDQ achieves up to 7.08$\times$ speedup and 90.23\% lower energy for mixed-precision GEMM, with only 0.127\,mm$^2$ area and 0.355\,W power overhead per DQB in a 12\,nm CMOS process. For end-to-end LLM inference, StreamDQ reduces latency by up to 54.68\% and improves decode throughput by up to 2.20$\times$.
LLM inference is increasingly constrained by accelerator memory capacity rather than compute throughput. This constraint is especially acute in single-accelerator and small-node inference systems, where limited on-package memory capacity restricts the size of deployable models. HBF is an emerging 3D-stacked NAND flash technology that provides multi-terabyte near-accelerator capacity, making it a promising capacity tier for storing LLM weights. However, existing HBF-based proposals face three adoption challenges: they (1) rely on coarse-grained static prefetching for LLM weights aiming to hide the microsecond-level read latency of the NAND flash device while maximizing HBF's read throughput, (2) expose NAND flash management tasks (e.g., refresh operations) to the accelerator-visible critical inference path, and (3) miss optimization opportunities to specialize and optimize the flash-management mechanisms to the workload behavior. Our goal is to design an efficient HBF substrate that integrates HBF as a memory-capacity tier alongside HBM while addressing these three challenges. To this end, we propose FLINT, a workload-driven HBF substrate for capacity-scalable LLM inference. FLINT introduces three mechanisms: (1) a hardware burst-buffer controller that dynamically coalesces and pipelines HBF reads aiming to utilize existing NAND flash buffers while sustaining high HBF bandwidth, (2) a phantom-plane refresh mechanism, which removes refresh from the critical inference path by moving refresh-related NAND flash operations outside the read foreground back via low-cost resource duplication, and (3) a read-only FTL, which replaces SSD-class support for arbitrary writes with a compact table that translates logical weight bursts to physical HBF locations.
Geraldo F. Oliveira, Arash Tavakkol, Xiang-Yu Zhu et al.· 0 citations
OasisKV is presented, a memory-centric LLM inference system design that alleviates HBM capacity pressure by decoupling full KV-cache storage from HBM during LLM decoding and observes that future important tokens can be predicted accurately in advance using lookahead tokens drafted by speculative decoding (SD).
Can Xiao, Sukmin Cho, Junbong We et al.· 0 citations
This paper presents FlashAttention-V, a blocked FlashAttention for scalable vector architectures that adapts efficiently from short to very long vectors by exploiting parallelism across attention heads, inter-head packing to enable efficient utilization of vector lengths beyond the head dimension, and improving vector register utilization and memory access locality.
The rapid growth of LLM inference has exposed critical bottlenecks in GPU memory capacity and data movement efficiency. When GPU memory is exhausted, modern inference frameworks rely on GPUDirect Storage (GDS) with PCIe P2P transfers to stage data—such as KV caches and intermediate tensors—between GPU and NVMe-based storage. NVMe storage remains a significant performance bottleneck, with state-of-the-art enterprise PCIe Gen5 SSDs delivering only 14 GB/s sequential reads and 7 GB/s writes per drive. In this work, we evaluate a CXL memory expander equipped with an on-device DMA controller as a high-performance staging tier in GDS-style data paths. Although contemporary GPUs do not natively participate in the CXL protocol, our design works around this gap: the device-side DMA controller drives PCIe peer-to-peer (P2P) transfers directly against GPU HBM, while CXL is used only on the host side for capacity expansion and device management. We develop a custom NIXL backend plugin to enable benchmarking of GPU-to-expander data movement, and use NIXLBench to characterize transfer performance across transfer sizes and request parallelism configurations. The expander achieves 51 GB/s read bandwidth (~80% of the PCIe Gen5 ×16 peak) and 33 GB/s write bandwidth—utilizing a substantially larger fraction of the PCIe Gen5 ×16 interconnect than leading Gen5 NVMe SSDs achieve on the same interconnect generation. The remaining ~1.5× read/write gap is consistent with PCIe transaction completion semantics and leaves room for further write-side optimization. Multi-threaded descriptor submission, a routine capability of modern inference frameworks, additionally lowers the block size at which peak bandwidth is reached from 8 MB to 256 KB, sustaining high bandwidth for fine-grained LLM inference workloads such as KV cache updates.
Veerasenareddy Burru, Pradeep Kumar Nalla, Alok Prasad· Proceedings of the ACM SIGCO...· 0 citations