H3-Attn: Attention-Efficient 3D DRAM PNM Processor with Hybrid Head Parallelism and Hierarchical SFU

Abstract

Edge-side LLM deployment, characterized by low-batch inference, has gained significance due to data privacy and personalization requirements. Attention is the key operator of LLM inference, involving massive KV cache access and full-context-aware softmax computation. To optimize KV cache size and softmax computation, various Attention mechanisms and the FlashAttention algorithm have been proposed. As 3D DRAM-based process-near-memory (PNM) presents a promising solution for low-batch LLM inference, processing these optimized Attention mechanisms may suffer from low utilization of 3D DRAM internal bandwidth due to the distributed memory access nature. Furthermore, existing Special Function Unit (SFU) architectures in 3D DRAM PNM for FlashAttention suffer from either severe communication bottlenecks in centralized designs or stringent area constraints in distributed implementations. To address these challenges, we propose H3-Attn, an Attention-efficient 3D DRAM PNM processor for low-batch LLM inference. H3-Attn features a hybrid head parallelism for Attention processing, whereby various optimized Attention mechanisms with spatial tiled FlashAttention can be flexibly enabled with fully leveraged 3D DRAM PNM bandwidth. A Hierarchical SFU (H-SFU) architecture with Multi-Head Instruction Interleaving (MHII) is also proposed, which significantly enhances softmax efficiency in FlashAttention by co-optimizing SFU area footprint and temporal utilization. Evaluations on various LLM models demonstrate that our proposed hybrid head parallelism achieves a 1.54-3.84× decoding speedup over the baseline Attention processing. The H-SFU with MHII delivers a 3.20× compute density improvement compared with prior SFU architectures for 3D DRAM PNM.