This work develops lightweight characterization methods to uncover per-chip compute topology and memory affinity and uses the discovered information to make existing fine-grained scheduling asymmetry-aware, considering not only how many resources are allocated but also which physical resources are assigned.
Abstract
Modern GPUs are no longer physically symmetric. Die scaling leads to both manufacturing-driven floorsweeping and cache and memory partitioning. The former creates chip-specific compute topologies, while the latter causes non-uniform memory access. These asymmetries are substantial. Topology-oblivious compute unit allocation can lead to up to 1.33x performance variation, while remote accesses increase HBM latency by up to 67% and nearly double L2 latency. However, these asymmetries are hidden behind the GPU's logical resource abstractions and can vary across chips. We develop lightweight characterization methods to uncover per-chip compute topology and memory affinity. We then use the discovered information to make existing fine-grained scheduling asymmetry-aware, considering not only how many resources are allocated but also which physical resources are assigned. Across full-GPU kernel execution, intra-application multiplexing, and inter-application co-location, asymmetry-aware scheduling improves mainstream kernels by up to 1.22x, multiplexed LLM inference by up to 14.3%, and avoids up to 1.33x performance variation.
Emerging artificial intelligence workloads, particularly Large Language Models (LLMs), place unprecedented demands on GPU memory systems. Consequently, architectural innovations increasingly target the memory system to alleviate these bottlenecks. However, existing works still rely on outdated memory hierarchy models t...
Zi-Wen Wang, Jia-Peng Zhou, Chang-Xi Liu et al.· 0 citations
Low latency is critical to user experience. As NVIDIA Blackwell and Rubin GPUs advance multi-die scaling, data placement and cross-die communication become increasingly important. However, the latency implications and underlying costs of cross-die communication and die-aware execution remain largely unexplored. In this...
Rocky Wang, Joyce Zhou, Ke Jiang et al.· 0 citations
Billion-scale Approximate Nearest Neighbor Search (ANNS) increasingly adopts a GPU-SSD architecture, where GPU fetches and processes fine-grained, SSD-resident data on-demand. However, since GPUs lack native support for I/O scheduling, existing ANNS systems are trapped in a severe throughput-latency tradeoff. They eith...
Pu-Qing Wu, Min-Hui Xie, Yi-Heng Tong et al.· Proceedings of the 2026 ACM...· 0 citations
Large language model (LLM) decode is dominated by short, predominantly memory-bound kernels that make bursty use of on-chip execution resources and memory bandwidth. This paper introduces SubGPU Pipelining, a fine-grained, intra-GPU spatial execution model that cuts an operator graph into ordered contiguous segments, b...
Muralidhar Andoorveedu, Marius Juston, Ting-Kai Liu et al.· 0 citations
Physical non-idealities in integrated circuit (IC) design, fabrication, and operation are concealed by conventional hardware abstractions, causing unexpected performance variations in kernels. For large-scale workloads, such as AI training and inference, performance unpredictability can lead to substantial losses. We i...
Supun Kuruppu, Pranav Dangi, Wei Li et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.