Aug 2026· ACM Transactions on Architecture and Code Optimization (TACO)· Vol 23, pp. 1-25· 1 citation· 14 references
Abstract
DRAM-based Processing-in-Memory (PIM) addresses the “memory wall” by executing computations directly inside main memory. However, memory interleaving and virtual memory limit contiguous data size visible to PIM units, constraining PIM task granularity. Fine-grained PIM tasks incur significant offloading overhead that negates PIM performance benefits. To mitigate this, existing PIM systems drastically isolate PIM memory or disable memory interleaving. These design choices, however, decrease the CPU memory bandwidth and introduce extra data transfer, leading to an additional “system memory wall” that degrades CPU performance and must be resolved to realize PIM’s full potential. In this work, we propose GUMPIM, a PIM system that allows interleaved CPU pages and non-interleaved PIM pages to coexist in a Unitary and Malleable memory space with Guaranteed PIM page allocation. GUMPIM enables zero-copy during PIM task offloading and maintains CPU memory bandwidth while ensuring low PIM offloading overhead. First, we propose a dual-track memory management mechanism consisting of independent page allocation and address translation for CPU and PIM pages. Second, we design GUMPIM interface hardware on PIM-enabled DRAMs to provide a dynamic address mapping for the different data layouts of CPU and PIM pages. Third, we propose a PIM-assisted page migration mechanism that transparently migrates pages while preserving CPU access bandwidth, thereby enabling guaranteed and accelerated PIM page allocation. GUMPIM requires no changes to commodity DRAM standards; all hardware modifications are limited to the DRAM side, ensuring full compatibility with existing CPUs and enabling immediate deployment on current HBMx- and LPDDRx-based PIM platforms. Our results show only <0.1% performance degradation for CPU workloads on GUMPIM, in contrast to the 25.8% degradation on PIM systems with memory interleaving turned off. For PIM workloads, GUMPIM reduces memory allocation and CPU-part computation times by 2.7× and 4.93×, respectively, yielding an end-to-end 2.3× speedup over a state-of-the-art baseline system.
Processing-in-Memory addresses the memory wall by co-locating computation with memory, but because real PIM hardware remains scarce, simulation is the primary way to explore the PIM design space. Yet existing PIM simulators each cover only part of that space: they typically model a single memory technology, fix processing elements at one level of the memory hierarchy, support a single execution model, and stop at the device boundary. We therefore present PIMID, an execution- and trace-driven full-system simulator that closes these gaps in one tool. PIMID supports both the shared-memory and message-passing execution models, running annotated parallel code in OpenMP and MPI side by side across eleven memory technologies (seven DRAM standards, SRAM, and three non-volatile memories); it places PEs anywhere from subarrays to logic dies, sweeps PE count and core-model fidelity, and prices the in-memory network per technology from measured congestion. Its single-process host-device co-simulation resolves an end-to-end time and energy breakdown (host preparation, device compute, and explicit boundary charges) that device-only tools cannot produce. Across the resulting dual-execution-model dataset, PIMID shows that the memory technology alone moves execution time by more than an order of magnitude and that the best host main memory is not the best PIM substrate; that regular kernels scale superlinearly with PE count as in-memory bandwidth co-scales with compute; that graph traversal under message-passing hits a collective-communication wall absent under shared memory; and that at full-system scope the offload trades time for energy only on the bandwidth-class memory: shared-memory offload saves energy on HBM3 while a 16-core host keeps every end-to-end time win. PIMID's plugin interfaces let new engines and models be added through standardized YAML specifications as PIM technology evolves.
Yuan He, Masaaki Kondo, Galen M. Shipman et al.· arXiv.org· 0 citations
Processing-in-Memory (PIM) promises to reduce data movement overhead by executing computation in or near memory, but its realized application speedup remains highly design-dependent. Non-offloadable host execution, host-PIM transfers, limited PIM capacity, and device programming latency can limit end-to-end speedup, making fast early-stage design-space exploration (DSE) essential. However, existing PIM evaluation methods remain limited: circuit- and device-level tools cannot capture these end-to-end PIM performance factors, while cycle-accurate simulation is too slow for iterative DSE. To address this gap, we present VIPER, a unified, lightweight, and architecture-aware performance evaluation framework for PIM DSE. VIPER profiles host execution once and combines the measured host behavior with a PIM-aware analytical engine that sweeps PIM-side parameters across candidate designs. It supports both Processing Near Memory (PNM) and Processing Using Memory (PUM) under task-offloading and data-triggered execution by capturing host-PIM transfer, array access, in-memory computation, device programming latency, and capacity-induced partitioning, providing rapid architecture-aware performance estimates for iterative DSE without repeated cycle-accurate simulation. We validate VIPER against a commercial UPMEM system and more than 400 cycle-accurate gem5 configurations. VIPER predicts the UPMEM offloading decision and break-even region a priori, and, with a refined transfer model, captures the measured peak-and-rolloff behavior with 12\% mean speedup error across the DPU sweep (6\% up to the 256-DPU peak). Against gem5, VIPER achieves less than 10\% error while reducing evaluation time from hours to under one minute. Case studies of UPMEM, ReRAM/FeFET crossbars, and IMCRYPTO show that architecture-aware DSE reveals key performance trade-offs that device-level evaluation misses.
Haoran Geng, T. Pereira, Xiaoyang Lu et al.· 0 citations
Processing-in-Memory (PIM) is promising for accelerating memory-bound decode attention, but attention acceleration alone is insufficient for production LLM serving, where engines dynamically allocate, populate, share, cache, and reclaim logical KV cache blocks. Supporting this lifecycle on commodity PIM requires efficient physical memory allocation, block-to-address mapping, and command generation. For the Value cache, these requirements create a fundamental conflict among GEMV efficiency, single-token write efficiency, and memory capacity: GEMV-optimized layouts scatter newly generated Value vectors across rows, making writes costly, while finer-grained memory sharing improves capacity utilization but fragments GEMV reductions. We present PATTON, a PIM runtime that integrates production LLM serving engines with commodity PIM. PATTON introduces hierarchical granule allocation: block-sized Key and Value granules map one-to-one to logical token blocks, fixing their physical placements and commands, while coarser granules group blocks for efficient GEMV execution and memory utilization. A Commit Zone stages partial Value blocks for efficient single-token writes before committing them to GEMV-optimized locations. PATTON tracks these placements to generate KV cache writes and QK-transpose/SV commands. Across attention execution and runtime-induced prefill recomputation, PATTON achieves an average 1.95x speedup and 4.83x higher energy efficiency over evaluated baselines, requires no PIM processing-unit modifications, and maintains a KV cache hit rate comparable to the native GPU KV cache in vLLM.
Hangyeol Kim, Sang-Hun Lee, Teokkyu Suh et al.· 0 citations
On-device mobile Large Language Model (LLM) inference is gaining significant attention. However, mobile devices operate in highly dynamic multitasking environments where users frequently switch between applications. This creates memory pressure, forcing LLM memory (model weights and KV cache) to be evicted by the operating system. When a new inference request arrives, the inference system must restore the evicted memory through slow storage reads or recompute the entire KV cache, severely degrading responsiveness. To address this, we present mzCache, an on-device LLM inference system with specialized memory management for multitasking environments. Under unpredictable memory pressure, mzCache elastically evicts LLM memory and leverages the unified memory of mobile SoCs to enable zero-wait inference on the GPU with concurrent CPU-side restoration. mzCache realizes this through restoration-oriented memory management: LLM memory is partitioned into fine-grained shared buffers to enable partial eviction and restoration with concurrent cross-processor access, while hybrid swap and backward-out eviction policies ensure low-latency restoration from any eviction state. Implemented on llama.cpp and deployed as an Android application, mzCache achieves 2.1-5.5$\times$ reduction in Time-to-First-Token compared to storage-backed partial offload and demonstrates its effectiveness in real multitasking scenarios.
Hong Yu, Minsung Kim, Jongseok Park et al.· 0 citations
With soaring DRAM prices, optimizing software for efficient memory use is becoming increasingly important. Essential to this end is the memory management component of the software's language runtime. This component requests page-sized memory from the kernel and subdivides it into smaller blocks to satisfy allocations performed by the software. Since DRAM was comparatively cheap for the preceding decade, contemporary prior work has focused on optimizing this component for performance. A key optimization is conducted on the free path where memory freed by the software is not directly returned to the kernel to satisfy future allocations and avoid costly context switches. In this work-in-progress paper, we illustrate the lost potential of user-level allocators for system-wide memory use. We show that—since the kernel manages memory at page-granularity—the user-level allocation scheme must be designed with page alignment in mind. We provide a classification for physical memory that is lost due to the mismatch of user-space and kernel-space allocation granularities. To empirically illustrate the potential of such an allocation scheme, we conduct experiments with the mallocng allocator where we identify potential savings of up to 75 percent.
Pasha Fistanto, Sören Tempel, Christian Dietrich· Proceedings of the 14th Work...· 0 citations
SmartNICs promise hardware offloading for network applications like traffic analysis and virtual host dispatching. However, due to memory bandwidth limitations, SmartNICs are unable to effectively accelerate applications like intrusion detection and deep packet inspection that require high-speed reassembly. We argue that this limitation is due to an architectural mismatch: variable-sized packets arrive out of order, but a SmartNIC's memory is a contiguous address space broken up into fixed-size pages. This mismatch forces multiple unnecessary copies to translate between the two layouts. We propose a new memory addressing scheme, flow-based addressing, which enables software to access the reassembled payloads of a flow's packets as a contiguous region in memory. Flow-based addressing allows software to easily operate on reassembled payloads without copies, dramatically reducing memory traffic. Eliminating copies entirely requires hardware that maps logical flow offsets to non-contiguous physical addresses and assembles misaligned payloads into contiguous cache lines at line rate. The new memory addressing scheme introduces three architectural mechanisms: a flow translation table, a private flow cache, and a specialized DMA engine that assembles unaligned, variable-length packets into contiguous cache lines. We describe a prototype design on an FPGA-based SmartNIC and show that, across synthetic microbenchmarks, university campus network traffic, and data center web-search traffic, flow-based addressing reduces DRAM bandwidth use by 77-83%, lowers execution time by 25-62%, and increases per-core throughput by 1.33-2.66×.
Agur Adams, H. Shim, Colin Drewes et al.· Proceedings of the ACM on Ne...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
