Jul 2026· IEEE International Conference on Cloud Computing· pp. 290-296· 0 citations· 30 references
Abstract
The rapid growth of Large Language Models has outpaced GPU memory capacity, making state-of-the-art models unfit for most GPUs. This necessitates evicting model parameters to CPU memory or disk, further away from the processing elements. In these scenarios, inference throughput is restricted by interconnect bandwidth, and while quantization schemes like FP8 reduce memory usage and enable larger models to fit smaller memories, they compromise generation quality and rely on specialized hardware support for efficient inference. To this end, we propose an inference pipeline that utilizes lossless entropy encoding (rANS) alleviating the bottleneck while maintaining standard floating-point portability or output precision. We demonstrate the potential of this approach by integrating the pipeline into vLLM, with a hardware-aware parallelization strategy. Our method increases the effective GPU memory capacity up to 30% in inference workloads with evicted weights, with no loss of quality and at the same time achieving speed up of 2.5–4.4%.
This work introduces custom Triton kernels with partial fusion and memory layout optimizations for both Monarch and BLAST, and delivers up to 3.76 × speedups and 3 × model size compression over PyTorch dense baselines using CUDA backend and compiler-level optimizations.
Pierre Abillama, Changwoo Lee, Juechu Dong et al.· IEEE International Symposium...· 0 citations
Beyond is presented, a drop-in runtime that integrates a smart offloading scheme to selectively identify and retain salient KV entries across continuous decoding sessions, together with a hybrid CPU–GPU attention mechanism for scalable inference.
Weishu Deng, Yujie Yang, Peiran Du et al.· IEEE International Symposium...· 1 citation
This study presents a cross-platform, multi-model empirical study, where several important observations are brought, including the contrastive effect of quantization under different hardware bottlenecks, along with a quantification of runtime delays caused by the lack of parallelism in the ARM architecture.
Subhransu Das, Jiaming Cheng, Swathi Vallabhajosyula et al.· Practice and Experience in A...· 0 citations
With the widespread adoption of large language models (LLMs), the demand for graphics processing units (GPUs)-essential for accelerating LLM training and inference- has increased significantly. This has led to rising procurement and operation costs, imposing a significant financial burden on research institutions and i...
Madoka Takato, Akira Hirai, Reika Kinoshita et al.· Annual International Compute...· 0 citations
This work evaluates a CXL memory expander equipped with an on-device DMA controller as a high-performance staging tier in GDS-style data paths, sustaining high bandwidth for fine-grained LLM inference workloads such as KV cache updates.
Adaptive Sequence Pipeline Parallel Offloading (SPPO) is proposed, a novel framework that optimizes memory and computational resource efficiency for long-sequence LLM training and develops an adaptive pipeline scheduling approach with a heuristic solver and multiplexed sequence partitioning to improve computational res...
Qiaoling Chen, Shenggui Li, Wei Gao et al.· International Conference on...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
