Aug 2026· Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication· pp. 696-713· 1 citation· 156 references
Computer Science
TL;DR
This work builds an ACN characterization framework that enables fine-grained, topology-aware probing of paths and links, and designs PingPoint, a lightweight utility for ACN-native profiling that uncovers fundamental ACN performance properties on multi-chiplet GPUs.
Abstract
Emerging chiplet-based accelerators introduce a new class of intrahost networks—the Accelerator Chiplet Network (ACN)—that links compute chiplets, IO chiplets, and memory modules and increasingly governs application performance. Yet ACN behavior remains largely opaque: existing tools overlook on-package communication and instead attribute overheads to compute or memory subsystems, while ACN-induced latency, bandwidth heterogeneity, and congestion are hard to observe due to proprietary microarchitectures, tight coupling with the execution pipeline, and complex mappings between application activity and hardware. To overcome this challenge, we build an ACN characterization framework that enables fine-grained, topology-aware probing of paths and links. We then use it to uncover fundamental ACN performance properties on multi-chiplet GPUs. Guided by these insights, we design PingPoint, a lightweight utility for ACN-native profiling. Our key insight is that modeling the ACN as a logical, hose-based graph with queueing abstractions, combined with in-situ software probing, makes systematic dissection of the otherwise opaque ACN possible. It injects latency and bandwidth probes while co-executing target kernels, captures cycle-level link- and path-granular distributions, and applies differential attribution to localize congestion to individual ACN links. Across diverse workloads and hardware, it exposes hidden bottlenecks, guides kernel placement and traffic shaping, quantifies the performance impact of ACN contention, and enables practical optimization with marginal overhead.
LLM inference has become an essential service, yet it imposes unprecedented demands on memory bandwidth, computational density, and communication efficiency. While IMC is a promising solution to the memory wall issue, the heterogeneous data dynamicity of LLM requires complementary resources to handle intermediate data generated during run-time. Furthermore, the massive number of parameters in LLM necessitates scale-up architectures where on-chip data movement is often the primary performance bottleneck. This article presents a hardware-software co-design framework that unifies distributed compute, memory, and communication into a seamless processing-communication fabric. On the hardware side, we propose a scalable architecture, named LEAP, that integrates IMC PE, NMC PE, and INC. This allows each hardware layer to execute specialized tasks: IMC for static weights, NMC for dynamic data, and INC for partial result reduction. On the software side, we introduce a partitioning, mapping, and scheduling framework optimized for key metrics in LLM serving, including throughput and latency. To address the distinct computational intensities of the prefill and decode phases, we present a prefill-decode disaggregation approach that dynamically reconfigures PE organizations to maximize resource utilization. Compared to commercial GPU platforms, the proposed architecture provides a throughput and an energy efficiency improvement of $\geq{}1.52\times$ and $24.91\times$, respectively.
The rapid growth of distributed machine learning systems shifts ML networks from the CPU/GPU-NIC to GPU-Fabrics architectures, creating an emerging need to measure and optimize NIC-less Scale-up networks. Yet, the disappearance of NICs removes key observability points and leaves existing measurement tools largely limited to coarse, application-layer telemetry. We present FabricPerf, a Scale-up network measurement toolkit that enables fine-grained packet-level timestamping and cross-layer hardware insights. FabricPerf uses a software-hardware co-design that integrates protocol-level probing and GPTP clock synchronization within GPU communication kernels with memory profiling of physical pipelines. We implement FabricPerf for the CUDA/NvLink ecosystem and validate its high precision and low overhead via extensive experiments on H100 and GB200 NVL72. Using FabricPerf, we explore Scale-up network dynamics under varied configurations, with two case studies that reveal critical bottlenecks: (i) channel imbalance in high P99 latency and (ii) memory pipeline inefficiency of ~0% LLC hit-rate. To mitigate these issues, we leverage work-stealing to rebalance channels by 46% and boost throughput by 17.5GB/s, and optimize LLC eviction priorities for memory flows to elevate the LLC read hit-rate to ~35%, yielding 16.07GB/s bandwidth improvement. We believe FabricPerf improves Scale-up network observability, laying a foundation for more complete and efficient network stacks. FabricPerf is open-sourced at github.com/open-neutrino/fabricperf.
Song-Ling Huang, Chenshu Wu· Conference on Applications,...· 0 citations
This work categorizes LLM kernel operands into three inter-workgroup sharing patterns and shows that the required optimization strategies differ across categories, from simple per-workgroup pinning to subgroup-aware co-scheduling, highlighting the need for placement-aware kernel programming and smarter architectural support for work and data locality in multi-partition GPUs.
Donghyeon Joo, Sooraj Puthoor, N. Jayasena et al.· arXiv.org· 1 citation
Mixture-of-experts inference introduces fine-grained, dynamically changing many-to-many communication for token dispatch and collection, making serving latency highly sensitive to the interconnect. Optical circuit switching (OCS) offers an optically transparent data plane, but its benefits for MoE depend critically on circuit reconfiguration latency, which has not been quantified for inference workloads. We develop a performance model and an OMNeT++ simulation methodology to quantify the reconfiguration-latency budget of an OCS-based switching fabric for MoE inference. Results reveal a sharp regime transition: nanosecond-scale reconfiguration preserves favorable latency, throughput, jitter, and task completion time, whereas microsecond-scale reconfiguration collapses throughput and inflates completion time by orders of magnitude. Trace-driven replay using measured DeepSeek-V3 inference communication traces collected from an 8-H20 GPU server confirms that the same latency regimes persist under measured MoE traffic, supporting the representativeness of the model-generated workload. With Tb/s link bandwidth and increasing oversubscription, the budget tightens to the tens-of-nanoseconds regime (e.g., <inline-formula><tex-math notation="LaTeX">$44.3 \,{\mathrm{ns}}$</tex-math></inline-formula> at <inline-formula><tex-math notation="LaTeX">$1.6 \,{\mathrm{Tbps}}$</tex-math></inline-formula> and <inline-formula><tex-math notation="LaTeX">$17.2 \,{\mathrm{ns}}$</tex-math></inline-formula> at <inline-formula><tex-math notation="LaTeX">$3.2 \,{\mathrm{Tbps}}$</tex-math></inline-formula> in the evaluated configuration). Finally, we demonstrate <inline-formula><tex-math notation="LaTeX">$43.4 \,\mathrm{n}\mathrm{s}$</tex-math></inline-formula> end-to-end circuit reconfiguration on a multi-endpoint prototype, validating feasibility at the implied timescales.
Shuo Li, Hua-Xi Gu, Yi-Xuan Hao et al.· Journal of Lightwave Technol...· 0 citations
SA-Scheduler is presented, a structure-derived bottleneck-aware scheduling framework for multitasking MCM-GPUs that determines chip placement without hardware modification or runtime bottleneck profiling, and provides a principled and scalable foundation for multitasking on future MCM-GPUs.
Tiejian Zhang, Guangda Zhang, Lu Wang et al.· ACM Transactions on Design A...· 0 citations
Graphics Processing Units (GPUs), due to their immense parallel processing capabilities, have become essential across various fields, including gaming and artificial intelligence. With significant advancements in GPU cores, GPU memory efficiency has lagged, resulting in bottlenecks that can limit workload efficiency. To bridge this gap, a deep understanding of GPU memory architectures, particularly Non-Uniform Memory Access (NUMA) mechanisms within L2 and DRAM, is essential for optimizing applications, designing new architectures, and building accurate simulators. However, the latest GPU hardware from vendors like NVIDIA and AMD is still a black-box, making it challenging for researchers to understand the details of their design. In this paper, we introduce DGNA, a methodology designed to unveil the NUMA architecture of the GPU memory hierarchy through microbenchmarking and data analysis. Specifically, we propose an approach to measuring the latency of L2 caches and DRAM without relying on the intrinsic instructions of the architecture and apply a Gaussian mixture model to filter out outliers and accurately determine latency distributions. We apply DGNA on NVIDIA’s A100 and H100 GPUs, revealing NUMA node architecture, SM-NUMA relationships, and NUMA-aware memory allocation strategies used to maintain cache coherence. To the best of our knowledge, this is the first paper to detail the NUMA architecture within the GPU memory subsystem.
Changxi Liu, Yun Chen, Trevor E. Carlson· ACM Transactions on Architec...· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
