Characterizing LLM Kernel Access and Memory Interaction in Multi-Partition NUMA GPUs
TL;DR
This work categorizes LLM kernel operands into three inter-workgroup sharing patterns and shows that the required optimization strategies differ across categories, from simple per-workgroup pinning to subgroup-aware co-scheduling, highlighting the need for placement-aware kernel programming and smarter architectural support for work and data locality in multi-partition GPUs.
Abstract
Large language model (LLM) workloads motivate multi-partition GPUs as a path to scaling compute and memory capacity, but their non-uniform memory access characteristics and inter-partition communication can amplify contention and degrade locality, leading to suboptimal kernel latency. To address this, we analyze performance-critical LLM kernel implementations spanning weight projection, mixture-of-experts, and attention variants of state-of-the-art serving engines to present a characterization of data access patterns in multi-partition GPUs. First, we introduce memory trace analysis methodology to derive workgroup-level data access and sharing behavior, then evaluate the locality implications on latency using a cycle-level simulator. Using these tools, we categorize LLM kernel operands into three inter-workgroup sharing patterns (global, partial, or private) and show that the required optimization strategies differ across categories, from simple per-workgroup pinning to subgroup-aware co-scheduling. Our findings highlight the need for placement-aware kernel programming and smarter architectural support for work and data locality in multi-partition GPUs.