Jul 2026· Practice and Experience in Advanced Research Computing· pp. 1-3· 0 citations· 5 references
Computer Science
TL;DR
Preliminary results show up to twice the throughput of rclone with substantially lower energy usage, and successful migration of projects with hundreds of millions of files that would be impractical with existing tools.
Abstract
Efficient data movement is a critical bottleneck in High Performance Computing (HPC) centers, especially for large-scale migrations across parallel and distributed storage systems. Existing tools such as rsync and rclone often underutilize network and metadata bandwidth, provide limited hard link preservation, and offer no guarantees on memory usage, limiting their effectiveness for HPC environments. Motivated by a large-scale migration of data at Partnership for an Advanced Computing Environment (PACE), we present copy2, a high-performance, resource-aware data transfer tool for modern HPC storage such as Lustre and VAST Data. copy2 features a scalable metadata crawler, inode-based hard link preservation, asynchronous high-throughput transfers, and bounded memory allocation. Preliminary results show up to twice the throughput of rclone with substantially lower energy usage, and successful migration of projects with hundreds of millions of files that would be impractical with existing tools. copy2 offers HPC administrators and researchers a robust, predictable, and efficient solution for large-scale data management.
Trace-based performance analysis provides essential insights for understanding and optimizing large-scale parallel applications. However, traces from applications running on tens of thousands of processes can easily exceed terabytes, far surpassing the memory capacity of typical computing nodes. Existing approaches either require expensive distributed analysis or cannot work on platforms with limited memory. To address these challenges, we present StreamTrace, a streaming trace analysis system that enables efficient analysis of massive traces on a single computing node with bounded memory consumption. StreamTrace introduces two key techniques: communication pattern-aware chunk partitioning that minimizes cross-chunk dependence, and dynamic priority-based chunk scheduling that reduces analysis waiting time by prioritizing frequently depended upon processes. Our evaluation on traces from applications with up to 8,192 processes demonstrates that StreamTrace achieves up to 3.48 × speedup on a single node over existing distributed systems. The ablation study shows the proposed technique can improve the performance over naive streaming implementations by up to 8.50 ×.
Yuyang Jin, Ji-Dong Zhai· Proceedings of the Internati...· 0 citations
Hierarchical Data Storage Systems (HSSs) provide a cost-effective architecture that balances capacity and performance through internal data migration. Prior work has primarily focused on optimizing individual migration tasks, either within or across HSS tiers, or on exploiting device bandwidth to improve overall throughput. These approaches treat migration tasks in isolation, and the performance implications of executing heterogeneous migration tasks concurrently remain largely unexplored despite their prevalence in real-world HSS deployments. The growing adoption of Direct Data Access (DDA) architectures, in which accelerators access storage without CPU mediation, further amplifies this problem by removing a natural bandwidth arbiter from the I/O path.
This paper presents an in-depth analysis of data migration behavior in commercial HSSs, uncovering substantial performance variability when multiple migration tasks execute concurrently. To mitigate this issue, we propose PASCAL, a system-level bandwidth orchestration framework that improves performance robustness in production-grade HSSs. Inspired by hydraulic systems, PASCAL adapts pressure/backpressure-style coordination to the multi-task migration setting: it treats each tier as a pressurized vessel and uses pressure gradients to allocate bandwidth across cache flush, tiering, garbage collection, and DDA flows. We evaluate PASCAL on a commercial OceanStor HSS across three hardware configurations and eight workloads spanning database, AI training, AI inference, and production traces. PASCAL achieves up to 20% higher throughput, 67% lower tail latency, and 79% reduced throughput jitter compared to local state-of-the-art controllers, while also stabilizing the performance jitter introduced by DDA architectures.
Ji Zhang, Li Liu, André Brinkmann et al.· Proceedings of the VLDB Endo...· 0 citations
Erasure coding provides efficient fault tolerance for large-scale distributed storage systems. However, its data repair process is well-known to be resource-intensive. We find that conventional host-centric, TCP-based repair architectures suffer from severe resource contention. Even in high-bandwidth networks, such interference slows repair operations and degrades service quality. To address this limitation, we propose DPUEC, a novel architecture that fully offloads the entire repair process to a Data Processing Unit (DPU). DPUEC leverages Remote Direct Memory Access (RDMA) to enable kernel-bypass data transfers and utilizes the onboard multi-core processors of DPU for highly parallel decoding. This design establishes a physically isolated data plane for repair traffic, completely bypassing host resources. Our prototype evaluation shows that DPUEC improves repair throughput by 133% - 181% over the state-of-the-art approach across different workloads. More importantly, DPUEC delivers true performance isolation. It eliminates any interference with foreground applications and reduces host CPU utilization on the recovery node to nearly zero.
Modern experimental instruments produce data faster than general-purpose file transfer interfaces can move it, so delivery to the computing infrastructure has become a bottleneck in the research process. At many universities and research institutes, moreover, the instruments that generate research data and the high-performance computing systems that analyze it are separated both geographically and organizationally, because each demands its own expertise and installation environment. Connecting the two seamlessly is a pressing challenge for data-driven science. This article presents RED-ONION, a high-speed disk-to-disk transfer system that connects research facilities, on campus and beyond, to a computing center. The system combines data transfer nodes, a dedicated high-bandwidth network, an all-flash parallel file system, and multi-threaded transfer software that parallelizes network transmission and storage access. The design targets the wire rate both along the entire path, from the read on the sender storage to the write on the receiver storage, and for a single file between one pair of nodes rather than only in aggregate over many files or nodes. We describe the end-to-end optimizations across the transfer software, the operating system, and the storage that this requires. We evaluate a prototype deployed over a 100 Gbps transpacific path between Atlanta and Tokyo with a 150 ms round-trip time, on which a single 1 TB file transfer reached 90 Gbps, delivering a terabyte in approximately 95 s. Moving a dataset of this size therefore becomes a routine step, and the computing center serves an instrument as if the two were co-located.
Keichi Takahashi, Hiroaki Kataoka, Takeo Hosomi et al.· 0 citations
Performance improvements in data-intensive Python applications have become more critical due to the increasing computational needs of modern analytics, machine learning, and large-scale data processing systems. Although the Python environment is enormous as well as flexible in development, frequent delay in execution, memory inefficiency and scalability problems are often encountered in many applications because of CPU demanding processes, over allocation of memory and I/O bottlenecks. In this research, we provide a systematic experimental technique to identify, classify, and resolve performance bottlenecks in large-scale Python applications. The approach we provide here unifies the profiling, benchmark based analysis, bottleneck detection, focused optimization, and quantitative validation into a single procedure. The proposed approach was tested on a transactional dataset of around 10 million records. We used performance profiling tools like cProfile, line_profiler and memory_profiler to identify computational bottlenecks, memory allocation inefficiencies and disk I/O latencies. Profiling findings were used to apply these optimization methods such as vectorization using NumPy and Pandas, multiprocessing, memory-efficient information management, and asynchronous I/O techniques. Experimental evaluation showed significant performance improvements, in particular decrease in execution time from 120 seconds to 50 seconds, reduction in peak memory use by roughly 38%, and significant gains in throughput and scalability under high workloads. The findings demonstrate that effective performance enhancement needs a systematic strategy that correlates bottleneck discovery, optimization selection along with validation rather than different tuning approaches. The proposed technique provides practical recommendations to improve computational performance, scalability, and resource consumption of data-intensive Python systems in production environments.
Madhurima Kommuru, Appala Nooka Kumar Doodala· International Journal of App...· 0 citations
WADO is presented, a distributed write-once-read-many (WORM) object-store runtime for data-centric workloads that closes this gap through three coordinated mechanisms: scope-centric processing, explicit pipeline decomposition, and interference-aware explicit parallelism.
Karim Youssef, Hariharan Devarajan, Nikoli Dryden et al.· International Conference on...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
