Experiments on multiple unseen scientific simulation datasets demonstrate that EVOLVE achieves substantially higher CRs than conventional compressors at comparable reconstruction quality, while delivering compression speeds that are orders of magnitude faster than INR-based methods, highlighting its promise as a strong alternative for compressing scientific data.
Abstract
Large-scale scientific simulations generate volumetric data at rates that far outpace advances in storage and network bandwidth, making effective lossy compression increasingly critical. However, conventional compressors often struggle to preserve fine structural details at high compression ratios (CRs), and implicit neural representations (INRs) require costly per-volume optimization and produce models with fixed CRs. To respond, we present EVOLVE, an autoencoder (AE)-based volume-compression framework that targets high CRs for offline compression, with three key contributions. First, we construct a large-scale cross-domain database of 6,376 volumes from 21 scientific simulations, curated via perceptual hashing to ensure diversity, enabling the optimized model to extract features that generalize across volumes within the covered scientific simulation domains. Second, we reexamine the design space of AE-based compressors and incorporate several macro- and micro-designs into a vanilla AE to develop EVOLVE, which substantially improves the expressive power and compression capability. Third, we develop a learnable gain mechanism with a three-stage training strategy to enable variable-rate encoding, allowing a single model to support continuous CR adjustment at inference time. Experiments on multiple unseen scientific simulation datasets demonstrate that EVOLVE achieves substantially higher CRs than conventional compressors at comparable reconstruction quality, while delivering compression speeds that are orders of magnitude faster than INR-based methods, highlighting its promise as a strong alternative for compressing scientific data. The code, model weights, and results are available on our project page at https://evolve-vis.github.io.
Feather is presented, a system that reduces this overhead by compressing intermediate activations before transmission and reconstructing them before downstream layers resume execution, consistently outperforming existing compression schemes, including linear autoencoder, PCA, and SVD.
Regan McDonald, M. Rego, Ertza Warraich et al.· Conference on Applications,...· 0 citations
This paper proposes a novel compression method that uses Singular Value Decomposition (SVD) informed by data examples that achieves substantial reductions in model size and computation with minimal impact on accuracy and demonstrates its applicability in compressing deep neural networks to a fraction of their original size.
Abdelfattah Toulaoui, H. Khalfi, I. Hafidi· Journal of Artificial Intell...· 0 citations
Cortado is compared with state-of-the-art approaches and it is found that it is the only progressive compressor that guarantees the evaluated error bounds on all tested inputs while achieving the highest throughputs, comparable reconstruction quality, and on-par compression ratios.
It is argued that replacing autoregressive LLMs with DLMs within the same compression framework could overcome the throughput bottleneck caused by their one-symbol-per-step limitation, and the newly proposed DLM-based framework advances the state of the art in lossless text compression.
Exascale simulations generate data far faster than it can be stored or analyzed, making efficient data reduction essential. Error-controlled lossy compression offers high compression ratios under user-specified error bounds, but the target tolerance must be fixed at compression time. Progressive compression relaxes this restriction, yet existing methods still rely on fixed refactoring strategies and do not fully exploit correlations among decomposed coefficients, limiting the efficiency of progressive retrieval. In this work, we present an adaptive progressive compression framework that improves retrieval efficiency for two common targets, namely error-bound and peak Signal-to-Noise ratios. Our contributions are fourfold. (1) We propose to leverage two complementary interpolation schemes for adaptive progressive compression toward different targets, and we optimize them to achieve high efficiency. (2) We propose coefficient decomposition, a novel method that exploits the commonly overlooked spatial correlations among decorrelated data, which further improves the efficiency. (3) We develop the adaptive progressive compression workflow with automatic selection of the best-fit refactoring pipeline and tailored optimizations. (4) We evaluate the proposed framework on five real-world scientific datasets against three state-of-the-art progressive compressors. Experimental results demonstrate that the proposed framework improves the compression ratio by up to $42.3\%$ under the same requested error tolerance and up to $92.5\%$ at the same PSNR, compared with the best-performing existing methods. When transferring $512$ GB of scientific data to remote sites, the framework delivers up to $1.26\times$ speedup in the end-to-end data transfer performance. Furthermore, our method achieves the highest visualization quality while retrieving the least amount of data from storage.
Soft context compression condenses a context into a few memory tokens that a frozen LLM consumes in place of the raw text, but existing compressors fix the compression ratio at training and inference: each deployed ratio requires a separately trained model, and the chosen ratio is applied uniformly to all inputs, whose actual needs vary drastically. We propose FlexComp, a method-agnostic framework that decouples the ratio from both training and deployment: Matryoshka-style training samples the memory budget $K$ per instance, turning one model into an any-ratio compressor, and the budget is then chosen per input by: (1) confidence-based cascade routing or (2) a lightweight learned $K$ predictor. Across ICAE, 500xCompressor, and SAC on MRQA, a single FlexComp model matches separately trained fixed-ratio specialists with minimal degradation. Cascade routing preserves over 98% of the mildest ratio's accuracy at up to 266x average compression; the $K$ predictor, in a single compression-decoding pass, reaches 158-236x within 0.7 F1 of the mildest ratio. At serving-scale batch sizes, the $K$ predictor cuts context KV cache by 50% and improves decoding throughput by 47%.
Kai-Yan Zhao, Zhong-Tao Miao, Akiko Aizawa et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
