Hierarchical Autoencoder-based Lossy Compression for Large-scale High-resolution Scientific Data

• URL: http://arxiv.org/abs/2307.04216v2
• Date: Mon, 6 May 2024 19:54:56 GMT
• Title: Hierarchical Autoencoder-based Lossy Compression for Large-scale High-resolution Scientific Data
• Authors: Hieu Le, Jian Tao,
• Abstract summary: This work presents a neural network that significantly compresses large-scale scientific data, but also maintains high reconstruction quality. The proposed model is tested with scientific benchmark data available publicly and applied to a large-scale high-resolution climate modeling data set. Our model achieves a compression ratio of 140 on several benchmark data sets without compromising the reconstruction quality.
• Score: 12.831138965071945
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Lossy compression has become an important technique to reduce data size in many domains. This type of compression is especially valuable for large-scale scientific data, whose size ranges up to several petabytes. Although Autoencoder-based models have been successfully leveraged to compress images and videos, such neural networks have not widely gained attention in the scientific data domain. Our work presents a neural network that not only significantly compresses large-scale scientific data, but also maintains high reconstruction quality. The proposed model is tested with scientific benchmark data available publicly and applied to a large-scale high-resolution climate modeling data set. Our model achieves a compression ratio of 140 on several benchmark data sets without compromising the reconstruction quality. 2D simulation data from the High-Resolution Community Earth System Model (CESM) Version 1.3 over 500 years are also being compressed with a compression ratio of 200 while the reconstruction error is negligible for scientific analysis.

Related papers

• Sparse $L^1$-Autoencoders for Scientific Data Compression [0.0]
  We introduce effective data compression methods by developing autoencoders using high dimensional latent spaces that are $L1$-regularized. We show how these information-rich latent spaces can be used to mitigate blurring and other artifacts to obtain highly effective data compression methods for scientific data.
  arXiv  Detail & Related papers  (2024-05-23T07:48:00Z)
• GWLZ: A Group-wise Learning-based Lossy Compression Framework for Scientific Data [14.92764869276237]
  We propose GWLZ, a novel group-wise learning-based lossy compression framework with multiple lightweight learnable enhancer models. We show that GWLZ significantly enhances the decompressed data reconstruction quality with negligible impact on the compression efficiency.
  arXiv  Detail & Related papers  (2024-04-20T21:12:53Z)
• Convolutional variational autoencoders for secure lossy image compression in remote sensing [47.75904906342974]
  This study investigates image compression based on convolutional variational autoencoders (CVAE) CVAEs have been demonstrated to outperform conventional compression methods such as JPEG2000 by a substantial margin on compression benchmark datasets.
  arXiv  Detail & Related papers  (2024-04-03T15:17:29Z)
• SRN-SZ: Deep Leaning-Based Scientific Error-bounded Lossy Compression with Super-resolution Neural Networks [13.706955134941385]
  We propose SRN-SZ, a deep learning-based scientific error-bounded lossy compressor. SRN-SZ applies the most advanced super-resolution network HAT for its compression. In experiments, SRN-SZ achieves up to 75% compression ratio improvements under the same error bound.
  arXiv  Detail & Related papers  (2023-09-07T22:15:32Z)
• Dataset Quantization [72.61936019738076]
  We present dataset quantization (DQ), a new framework to compress large-scale datasets into small subsets. DQ is the first method that can successfully distill large-scale datasets such as ImageNet-1k with a state-of-the-art compression ratio.
  arXiv  Detail & Related papers  (2023-08-21T07:24:29Z)
• Scalable Hybrid Learning Techniques for Scientific Data Compression [6.803722400888276]
  Scientists require compression techniques that accurately preserve derived quantities of interest (QoIs) This paper presents a physics-informed compression technique implemented as an end-to-end, scalable, GPU-based pipeline for data compression.
  arXiv  Detail & Related papers  (2022-12-21T03:00:18Z)
• COIN++: Data Agnostic Neural Compression [55.27113889737545]
  COIN++ is a neural compression framework that seamlessly handles a wide range of data modalities. We demonstrate the effectiveness of our method by compressing various data modalities.
  arXiv  Detail & Related papers  (2022-01-30T20:12:04Z)
• Exploring Autoencoder-based Error-bounded Compression for Scientific Data [14.724393511470225]
  We develop an error-bounded autoencoder-based framework in terms of the SZ model. We optimize the compression quality for the main stages in our designed AE-based error-bounded compression framework.
  arXiv  Detail & Related papers  (2021-05-25T07:53:32Z)
• Towards Compact CNNs via Collaborative Compression [166.86915086497433]
  We propose a Collaborative Compression scheme, which joints channel pruning and tensor decomposition to compress CNN models. We achieve 52.9% FLOPs reduction by removing 48.4% parameters on ResNet-50 with only a Top-1 accuracy drop of 0.56% on ImageNet 2012.
  arXiv  Detail & Related papers  (2021-05-24T12:07:38Z)
• ScaleCom: Scalable Sparsified Gradient Compression for Communication-Efficient Distributed Training [74.43625662170284]
  Large-scale distributed training of Deep Neural Networks (DNNs) on state-of-the-art platforms is expected to be severely communication constrained. We propose a new compression technique that leverages similarity in the gradient distribution amongst learners to provide significantly improved scalability. We experimentally demonstrate that ScaleCom has small overheads, directly reduces gradient traffic and provides high compression rates (65-400X) and excellent scalability (up to 64 learners and 8-12X larger batch sizes over standard training) without significant accuracy loss.
  arXiv  Detail & Related papers  (2021-04-21T02:22:10Z)
• Analyzing and Mitigating JPEG Compression Defects in Deep Learning [69.04777875711646]
  We present a unified study of the effects of JPEG compression on a range of common tasks and datasets. We show that there is a significant penalty on common performance metrics for high compression.
  arXiv  Detail & Related papers  (2020-11-17T20:32:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.