Can Deep Neural Networks Improve Compression of Very Large Scientific Data?

• URL: http://arxiv.org/abs/2606.14353v1
• Date: Fri, 12 Jun 2026 11:22:26 GMT
• Title: Can Deep Neural Networks Improve Compression of Very Large Scientific Data?
• Abstract summary: We present a framework that integrates spatial and temporal deep learning models into a conventional error-bounded compression pipeline.<n>We evaluate three distinct ML predictors: a VAEformer-based (CRA5), a graph neural network forecaster (GraphCast), and a vision-transformer forecaster (Aurora)<n>Our evaluation over approximately 1.7 TB of data reveals a surprising result: although ML predictors generate more accurate predictions, they do not improve overall dataset-level compression ratio.
• Score: 4.2414540423650795
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Error-bounded lossy compression is a fundamental technique for managing the rapidly growing volumes of scientific data produced by modern simulations and observational instruments. Most state-of-the-art-compressors follow a prediction-residual paradigm, where compression effectiveness depends on the quality of the predictor: more accurate predictions generate smaller residuals that are easier to compress. This observation raises a question: can modern machine learning models serve as superior predictors for scientific data compression? Answering this question directly is challenging because developing compression-specific ML predictors requires substantial resources. Instead, we leverage the climate domain where highly accurate pretrained weather forecasting foundation models already exist, making them an ideal testbed. We present a framework that integrates spatial and temporal deep learning models into a conventional error-bounded compression pipeline. The framework supports auto-regressive forecasting models and avoids error accumulation. Using ERA5 climate data as a representative large-scale scientific dataset, we evaluate three distinct ML predictors: a VAEformer-based codec (CRA5), a graph neural network forecaster (GraphCast), and a vision-transformer forecaster (Aurora), against the state-of-the-art compressor SZ3.1 under identical quantization and entropy-coding backends. Our evaluation over approximately 1.7 TB of data reveals a surprising result: although ML predictors generate more accurate predictions and can improve reconstruction quality by up to 91% while achieving up to 9.6x higher compression ratios for highly predictable variables, they do not improve overall dataset-level compression ratio. We show that prediction accuracy alone is insufficient: the spatial structure of the resulting residuals plays a decisive role in entropy coding efficiency.

Related papers

• DeepCQ: General-Purpose Deep-Surrogate Framework for Lossy Compression Quality Prediction [4.634179787231294]
  We present a general-purpose deep-surrogate framework for lossy compression quality prediction (DeepCQ)<n>Our results highlight the framework's exceptional predictive accuracy, with prediction errors generally under 10% across most settings.
  arXiv  Detail & Related papers  (2025-12-24T21:46:17Z)
• LLMComp: A Language Modeling Paradigm for Error-Bounded Scientific Data Compression (Technical Report) [4.2414540423650795]
  LLMCOMP is a lossy compression paradigm that leverages decoder-only large language models to model scientific data.<n>It consistently outperforms state-of-the-art compressors, achieving up to 30% higher compression ratios under strict error bounds.
  arXiv  Detail & Related papers  (2025-10-24T05:41:04Z)
• Unified Scaling Laws for Compressed Representations [69.72517034565467]
  We investigate whether a unified scaling framework can accurately predict model performance when training occurs over various compressed representations.<n>Our main finding is demonstrating both theoretically and empirically that there exists a simple "capacity" metric.<n>We extend our formulation to directly compare the accuracy potential of different compressed formats, and to derive better algorithms for training over sparse-quantized formats.
  arXiv  Detail & Related papers  (2025-06-02T16:52:51Z)
• CompGS++: Compressed Gaussian Splatting for Static and Dynamic Scene Representation [60.712165339762116]
  CompGS++ is a novel framework that leverages compact Gaussian primitives to achieve accurate 3D modeling.<n>Our design is based on the principle of eliminating redundancy both between and within primitives.<n>Our implementation will be made publicly available on GitHub to facilitate further research.
  arXiv  Detail & Related papers  (2025-04-17T15:33:01Z)
• Variable Rate Neural Compression for Sparse Detector Data [9.331686712558144]
  We propose a novel approach for TPC data compression via key-point identification facilitated by sparse convolution. BCAE-VS achieves a $75%$ improvement in reconstruction accuracy with a $10%$ increase in compression ratio over the previous state-of-the-art model.
  arXiv  Detail & Related papers  (2024-11-18T17:15:35Z)
• NeurLZ: An Online Neural Learning-Based Method to Enhance Scientific Lossy Compression [34.30562110131907]
  NeurLZ is a neural method designed to enhance lossy compression by integrating online learning, cross-field learning, and robust error regulation.<n>During the first five learning epochs, NeurLZ achieves an 89% bit rate reduction, with further optimization yielding up to around 94% reduction at equivalent distortion.
  arXiv  Detail & Related papers  (2024-09-09T16:48:09Z)
• CRA5: Extreme Compression of ERA5 for Portable Global Climate and Weather Research via an Efficient Variational Transformer [22.68937280154092]
  We introduce an efficient neural, the Variational Autoencoder Transformer (VAEformer), for extreme compression of climate data. VAEformer outperforms existing state-of-the-art compression methods in the context of climate data. Experiments show that global weather forecasting models trained on the compact CRA5 dataset achieve forecasting accuracy comparable to the model trained on the original dataset.
  arXiv  Detail & Related papers  (2024-05-06T11:30:55Z)
• CogDPM: Diffusion Probabilistic Models via Cognitive Predictive Coding [62.075029712357]
  This work introduces the Cognitive Diffusion Probabilistic Models (CogDPM) CogDPM features a precision estimation method based on the hierarchical sampling capabilities of diffusion models and weight the guidance with precision weights estimated by the inherent property of diffusion models. We apply CogDPM to real-world prediction tasks using the United Kindom precipitation and surface wind datasets.
  arXiv  Detail & Related papers  (2024-05-03T15:54:50Z)
• Compression of Structured Data with Autoencoders: Provable Benefit of Nonlinearities and Depth [83.15263499262824]
  We prove that gradient descent converges to a solution that completely disregards the sparse structure of the input. We show how to improve upon Gaussian performance for the compression of sparse data by adding a denoising function to a shallow architecture. We validate our findings on image datasets, such as CIFAR-10 and MNIST.
  arXiv  Detail & Related papers  (2024-02-07T16:32:29Z)
• Learning Sample Difficulty from Pre-trained Models for Reliable Prediction [55.77136037458667]
  We propose to utilize large-scale pre-trained models to guide downstream model training with sample difficulty-aware entropy regularization. We simultaneously improve accuracy and uncertainty calibration across challenging benchmarks.
  arXiv  Detail & Related papers  (2023-04-20T07:29:23Z)
• A Physics-Informed Vector Quantized Autoencoder for Data Compression of Turbulent Flow [28.992515947961593]
  We apply a physics-informed Deep Learning technique based on vector quantization to generate a low-dimensional representation of data from turbulent flows. The accuracy of the model is assessed using statistical, comparison-based similarity and physics-based metrics. Our model can offer CR $=85$ with a mean square error (MSE) of $O(10-3)$, and predictions that faithfully reproduce the statistics of the flow, except at the very smallest scales.
  arXiv  Detail & Related papers  (2022-01-10T19:55:50Z)
• Learning to Predict Trustworthiness with Steep Slope Loss [69.40817968905495]
  We study the problem of predicting trustworthiness on real-world large-scale datasets. We observe that the trustworthiness predictors trained with prior-art loss functions are prone to view both correct predictions and incorrect predictions to be trustworthy. We propose a novel steep slope loss to separate the features w.r.t. correct predictions from the ones w.r.t. incorrect predictions by two slide-like curves that oppose each other.
  arXiv  Detail & Related papers  (2021-09-30T19:19:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.