Skip to content
Log in
Sponsored Artificial Intelligence: A Modern Approach, Global Edition Artificial Intelligence: A Modern Approach, Global Edition

Author

Yuquan Ding

View on Semantic Scholar

2 papers indexed here

We haven’t gathered this author’s papers yet. Follow them and we’ll fetch their work.

Not the right person? Other researchers publish under this name.

Open access Aug 2026

GPU and CPU Memory Co-Optimization in Heterogeneous Pipeline Parallelism for Efficient Large Language Model Fine-Tuning on Commodity Servers

Tiny-Pipe comprises a holistic layer packing method that simultaneously reduces GPU memory footprint and improves training performance, an active CPU memory management that alleviates CPU memory pressure by eliminating redundant parameters, and a layer-wise runtime swapping strategy that further enhances overall performance.

Yu-Quan Ding, Jie Shao · 0 citations
Open access

GPU and CPU Memory Co-Optimization in Heterogeneous Pipeline Parallelism for Efficient Large Language Model Fine-Tuning on Commodity Servers

To train or fine-tune large language models with insufficient GPU memory, heterogeneous parallel training methods utilize aggregated GPU memory and offload tensors to CPU DRAM or SSD. However, these methods lack effective simultaneous management of GPU and CPU memory, creating a critical bottleneck on resource-constrained commodity servers since insufficient capacity in either component leads to training failure. Moreover, existing methods incur excessive GPU memory usage without corresponding performance gains. To address these limitations, we propose a heterogeneous pipeline parallelism scheme named Tiny-Pipe that efficiently utilizes both GPU and CPU memory while maintaining comparable or superior performance. Tiny-Pipe comprises three key components: (1) a holistic layer packing method that simultaneously reduces GPU memory footprint and improves training performance, (2) active CPU memory management that alleviates CPU memory pressure by eliminating redundant parameters, and (3) a layer-wise runtime swapping strategy that further enhances overall performance. Experimental results demonstrate that our approach achieves (1) the smallest GPU and CPU memory footprint across all cases and (2) the broadest training coverage—successfully training all model configurations—while (3) maintaining optimal performance in most scenarios.

Yuquan Ding, Jie Shao · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.