Lossless Compression Performance for PETRA III Datasets
TL;DR
This first comprehensive study of data heterogeneity and lossless general-purpose compression performance for representative datasets from the PETRA III synchrotron radiation source provides a quantitative basis for future archival and storage decisions at PETRA III, its future successor, PETRA IV, and other large-scale scientific facilities.
Abstract
Large-scale research facilities increasingly face the challenge of managing rapidly growing data volumes while maintaining sustainable archival infrastructures. We present the first comprehensive study of data heterogeneity and lossless general-purpose compression performance for representative datasets from the PETRA III synchrotron radiation source. Our corpus comprises more than 212 TiB of raw and processed data from ten experiments spanning multiple beamlines, detector systems, and scientific workflows. We observe substantial heterogeneity both between and within experiments, resulting in compression ratios that vary by more than two orders of magnitude across datasets. Evaluating nine widely used lossless compression tools, we find that Zstandard and LZ4 consistently occupy the high-throughput region of the Pareto front, whereas ZPAQ achieves the highest compression ratios. Furthermore, heterogeneous compression strategies that adapt compressor choice to the underlying file category outperform uniform compression policies. Extrapolating from the benchmarked datasets to the full PETRA III non-tape storage system, we estimate achievable compression ratios ranging from approximately 1.6 at $\sim$900 MiB/s throughput to 2.1 at $\sim$2 MiB/s. These results provide a quantitative basis for future archival and storage decisions at PETRA III, its future successor, PETRA IV, and other large-scale scientific facilities.