· ACM Transactions on Architecture and Code Optimization· Vol 23, pp. 1-25· 0 citations
Abstract
To train or fine-tune large language models with insufficient GPU memory, heterogeneous parallel training methods utilize aggregated GPU memory and offload tensors to CPU DRAM or SSD. However, these methods lack effective simultaneous management of GPU and CPU memory, creating a critical bottleneck on resource-constrained commodity servers since insufficient capacity in either component leads to training failure. Moreover, existing methods incur excessive GPU memory usage without corresponding performance gains. To address these limitations, we propose a heterogeneous pipeline parallelism scheme named Tiny-Pipe that efficiently utilizes both GPU and CPU memory while maintaining comparable or superior performance. Tiny-Pipe comprises three key components: (1) a holistic layer packing method that simultaneously reduces GPU memory footprint and improves training performance, (2) active CPU memory management that alleviates CPU memory pressure by eliminating redundant parameters, and (3) a layer-wise runtime swapping strategy that further enhances overall performance. Experimental results demonstrate that our approach achieves (1) the smallest GPU and CPU memory footprint across all cases and (2) the broadest training coverage—successfully training all model configurations—while (3) maintaining optimal performance in most scenarios.
Deep neural networks (DNNs) with billions of parameters power many important applications, but their training is fundamentally constrained by the limited on-chip memory of GPUs. This memory wall forces training to rely on distributed execution or memory offloading, both of which introduce substantial inefficiencies. Existing offloading techniques can scale model size but often incur severe throughput degradation, while conventional distributed training suffers from poor hardware utilization due to limited cross-device bandwidth, leaving accelerator resources underused. We present SynergyScale, a software framework that addresses the GPU memory wall by jointly optimizing memory usage and computation efficiency. SynergyScale combines dynamic hierarchical memory offloading across GPUs, CPU, and secondary storage tiers to reduce GPU memory pressure with fine-grained multi-stream task partitioning that takes advantage of the concurrent execution capabilities of modern GPUs. The framework automatically selects offloading parameters that balance data movement overhead with parallel execution efficiency. Experimental results show that SynergyScale can train a 114.41B-parameter model on a single 80 GB NVIDIA A100 GPU and a 547.77B-parameter model on eight A100 GPUs, supporting models up to 29.5× larger than those enabled by existing approaches. By co-optimizing offloading and computation, SynergyScale achieves up to 4.5× higher training throughput than offloading-only methods, without altering training semantics, making billion-scale model training both efficient and cost-effective.
Xiaoyang Sun, Jie Xu, Zheng Wang· IEEE Transactions on Paralle...· 0 citations
This work proposes FIBER, a new architecture that extends the GPU SIMT (single instruction, multiple thread) model, and extends the ISA, microarchitecture, and compiler to realize shared-register addressing, conflict-free operand delivery, and fiber-based program mapping.
Zihan Liu, Jingwen Leng, Yangjie Zhou et al.· 0 citations
LazyTrain is proposed, an optimization layer over a layer-streaming executor that formulates checkpoint selection, activation placement, recomputation, and CPU-GPU-NVMe communication overlap as a mixed-integer scheduling problem, then executes the solved policy during training.
Xiao-Jun Wu, Ce-Hao Yang, Honghao Liu et al.· 0 citations
Sparse Matrix-Matrix Multiplication (SpMM) is a fundamental operation in scientific computing and artificial intelligence. However, the inherent sparsity and irregularity of real-world datasets present significant challenges for developing high-performance SpMM kernels on modern GPUs. Existing approaches typically focus on a single type of compute unit, leaving the potential for cooperative parallelism between heterogeneous GPU cores largely under-explored. This paper introduces CoTC-SpMM, a cooperative Tensor–CUDA cores scheme for efficient SpMM on GPUs. Specifically, we first propose the HTC format to partition sparse matrices into dense and sparse components, enabling specialized kernels to leverage the distinct advantages of different computing units and maximize hardware utilization. Moreover, we implement several low-level runtime optimizations, including 1-D resource mapping for load balancing, software pipelining to hide memory latency, and PTX-level instruction tuning to enhance SpMM throughput. Experimental results on NVIDIA A100 and H800 GPUs demonstrate that CoTC-SpMM achieves substantial performance speedups over state-of-the-art implementations.
Qi Du, Shengle Lin, Yuedan Chen et al.· Proceedings of the Internati...· 0 citations
This work proposes SAI, a mechanism that virtualizes shared memory into the L2 cache to improve GPU performance for AI applications and introduces an L2 cache management strategy that integrates associativity-based virtual page allocation and a replacement information table, reducing page-swapping overhead while preserving L2 cache performance.
Hanqing Li, Tie-Jun Li, Sheng Ma et al.· ACM Transactions on Design A...· 0 citations
Heterogeneous architectures featuring CPUs and GPUs in one system are increasingly adopted for high-performance data processing, yet interconnect bandwidth and memory capacity remain primary bottlenecks on the GPU side. While high-end solutions like NVIDIA Grace Hopper mitigate these issues via specialized interconnects, their high cost limits widespread adoption. We investigate the memory capabilities of the NVIDIA DGX Spark, a novel platform featuring a unified memory architecture where DDR memory is located on the CPU and is fully accessible from the GPU. We analyze the performance and tuning of such a system and explore how data processing workloads can be best run on such shared memory architectures. The demonstration will showcase how memory is allocated, the performance implications of different configurations, the results of running a data analytics benchmark, and the tools used to run the benchmarks, insert instrumentation, and analyze the results.
Silvia R. Alcaraz, S. Hepkema, Vasilis Mageirakos et al.· Proceedings of the 4th Works...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
