Conference
Jul 2026
Reducing Memory Requirements of LLM Inference Through Online rANS Decompression
The rapid growth of Large Language Models has outpaced GPU memory capacity, making state-of-the-art models unfit for most GPUs. This necessitates evicting model parameters to CPU memory or disk, further away from the processing elements. In these scenarios, inference throughput is restricted by interconnect bandwidth,...
Väinö-Waltteri Granat, Jakub Žádník, M. Mäkitalo et al.
· IEEE International Conferenc... · 0 citations