TL;DR
This work investigates lightweight BF16 weight compression schemes tailored for PIM-based LLM inference by focusing on exponent-oriented compression methods that exploit the locality and redundancy present in BF16 exponent fields.
The rapid growth of Large Language Models has outpaced GPU memory capacity, making state-of-the-art models unfit for most GPUs. This necessitates evicting model parameters to CPU memory or disk, further away from the processing elements. In these scenarios, inference throughput is restricted by interconnect bandwidth,...
FLINT is proposed, a workload-driven HBF substrate for capacity-scalable LLM inference that integrates HBF as a memory-capacity tier alongside HBM while addressing three adoption challenges.
StreamDQ is proposed, a lightweight architectural enhancement that enables on-the-fly dequantization in the memory subsystem for high-throughput, large-batch LLM inference and reduces latency and improves decode throughput for end-to-end LLM inference.
SLIM (Saturation-Aware Lightweight Performance Model), a semi-analytical model that predicts LLM inference throughput and latency from analytical formulations of Transformer computation and memory traffic, is introduced, which outperforms representative performance-modeling baselines while successfully generalizing to...
Design rules and a reproducible evaluation protocol are contributed that jointly report quality, memory, and end-to-end speed, and a foundation for automated pipeline search under realistic single-GPU constraints is provided.
This paper presents FlashAttention-V, a blocked FlashAttention for scalable vector architectures that adapts efficiently from short to very long vectors by exploiting parallelism across attention heads, inter-head packing to enable efficient utilization of vector lengths beyond the head dimension, and improving vector...
We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.
