Experience with NVIDIA GPUDirect Storage (GDS) in Academic HPC Environments: Challenges, Pitfalls, and Practical Limitations
TL;DR
The results show that GDS performance depends heavily on file sizes, access patterns, and storage backends, and the path from benchmark to production is longer and more expensive than marketing materials suggest.
Abstract
The use of general-purpose GPUs has become essential in modern computing, particularly for workflows that are throughput-bound, e.g., deep learning workflows, LLMs, image segmentation, etc. Because GPUs function as coprocessors, data transfer between the CPU and GPU is constrained by the bandwidth of the interconnect. To mitigate this bottleneck, GPUDirect Storage (GDS) was introduced by Nvidia to enable more efficient data movement and improve overall system performance. GDS allows direct data transfers between storage and GPU memory, bypassing the CPU entirely. Vendors claim significant bandwidth improvements with minimal code changes. In this work, we report our experience deploying GDS on OSCAR, Brown University’s heterogeneous HPC cluster. We deployed and tested GDS across three storage configurations: VAST Data (pNFS over 200G HDR InfiniBand), IBM Spectrum Scale (GPFS over NDR InfiniBand), and local NVMe drives on DGX systems. We executed benchmarks both vendor-provided and a production workload using a Scientific Machine Learning (SciML) benchmark. Our results show that GDS performance depends heavily on file sizes, access patterns, and storage backends. The vendor benchmarks showed improvements in specific scenarios, but these gains did not always translate to the SciML benchmarks. Beyond the performance results, the deployment itself consumed months of staff time, multiple support tickets across NVIDIA, VAST, and IBM, and significant unplanned expenditure on dedicated optical components and cables that vendor planning documents never mentioned. The gains, where they existed, were modest. More critically, application-level support remains immature: PyTorch lists GDS integration as experimental, and at the time of writing, their own tutorial code has been removed from the documentation website. We discussed this issue with PyTorch developers via GitHub, but no clear solution was provided. The feature is currently labeled as experimental and does not appear to be under active development. By sharing our experience, we hope to give other HPC centers realistic expectations for GDS deployments. The technology works, but the path from benchmark to production is longer and more expensive than marketing materials suggest.