Towards Extreme Image Compression with Latent Feature Guidance and Diffusion Prior

• URL: http://arxiv.org/abs/2404.18820v4
• Date: Wed, 4 Sep 2024 03:35:58 GMT
• Title: Towards Extreme Image Compression with Latent Feature Guidance and Diffusion Prior
• Authors: Zhiyuan Li, Yanhui Zhou, Hao Wei, Chenyang Ge, Jingwen Jiang,
• Abstract summary: We propose a novel two-stage extreme image compression framework that exploits the powerful generative capability of pre-trained diffusion models. Our method significantly outperforms state-of-the-art approaches in terms of visual performance at extremely lows.
• Score: 8.772652777234315
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Image compression at extremely low bitrates (below 0.1 bits per pixel (bpp)) is a significant challenge due to substantial information loss. In this work, we propose a novel two-stage extreme image compression framework that exploits the powerful generative capability of pre-trained diffusion models to achieve realistic image reconstruction at extremely low bitrates. In the first stage, we treat the latent representation of images in the diffusion space as guidance, employing a VAE-based compression approach to compress images and initially decode the compressed information into content variables. The second stage leverages pre-trained stable diffusion to reconstruct images under the guidance of content variables. Specifically, we introduce a small control module to inject content information while keeping the stable diffusion model fixed to maintain its generative capability. Furthermore, we design a space alignment loss to force the content variables to align with the diffusion space and provide the necessary constraints for optimization. Extensive experiments demonstrate that our method significantly outperforms state-of-the-art approaches in terms of visual performance at extremely low bitrates. The source code and trained models are available at https://github.com/huai-chang/DiffEIC.

Related papers

• Map-Assisted Remote-Sensing Image Compression at Extremely Low Bitrates [47.47031054057152]
  Generative models have been explored to compress RS images into extremely low-bitrate streams. These generative models struggle to reconstruct visually plausible images due to the highly ill-posed nature of extremely low-bitrate image compression. We propose an image compression framework that utilizes a pre-trained diffusion model with powerful natural image priors to achieve high-realism reconstructions.
  arXiv  Detail & Related papers  (2024-09-03T14:29:54Z)
• Zero-Shot Image Compression with Diffusion-Based Posterior Sampling [34.50287066865267]
  This work addresses the gap by harnessing the image prior learned by existing pre-trained diffusion models for solving the task of lossy image compression. Our method, PSC (Posterior Sampling-based Compression), utilizes zero-shot diffusion-based posterior samplers. PSC achieves competitive results compared to established methods, paving the way for further exploration of pre-trained diffusion models and posterior samplers for image compression.
  arXiv  Detail & Related papers  (2024-07-13T14:24:22Z)
• HybridFlow: Infusing Continuity into Masked Codebook for Extreme Low-Bitrate Image Compression [51.04820313355164]
  HyrbidFlow combines the continuous-feature-based and codebook-based streams to achieve both high perceptual quality and high fidelity under extreme lows. Experimental results demonstrate superior performance across several datasets under extremely lows.
  arXiv  Detail & Related papers  (2024-04-20T13:19:08Z)
• Correcting Diffusion-Based Perceptual Image Compression with Privileged End-to-End Decoder [49.01721042973929]
  This paper presents a diffusion-based image compression method that employs a privileged end-to-end decoder model as correction. Experiments demonstrate the superiority of our method in both distortion and perception compared with previous perceptual compression methods.
  arXiv  Detail & Related papers  (2024-04-07T10:57:54Z)
• Progressive Learning with Visual Prompt Tuning for Variable-Rate Image Compression [60.689646881479064]
  We propose a progressive learning paradigm for transformer-based variable-rate image compression. Inspired by visual prompt tuning, we use LPM to extract prompts for input images and hidden features at the encoder side and decoder side, respectively. Our model outperforms all current variable image methods in terms of rate-distortion performance and approaches the state-of-the-art fixed image compression methods trained from scratch.
  arXiv  Detail & Related papers  (2023-11-23T08:29:32Z)
• You Can Mask More For Extremely Low-Bitrate Image Compression [80.7692466922499]
  Learned image compression (LIC) methods have experienced significant progress during recent years. LIC methods fail to explicitly explore the image structure and texture components crucial for image compression. We present DA-Mask that samples visible patches based on the structure and texture of original images. We propose a simple yet effective masked compression model (MCM), the first framework that unifies LIC and LIC end-to-end for extremely low-bitrate compression.
  arXiv  Detail & Related papers  (2023-06-27T15:36:22Z)
• Extreme Generative Image Compression by Learning Text Embedding from Diffusion Models [13.894251782142584]
  We propose a generative image compression method that demonstrates the potential of saving an image as a short text embedding. Our method outperforms other state-of-the-art deep learning methods in terms of both perceptual quality and diversity.
  arXiv  Detail & Related papers  (2022-11-14T22:54:19Z)
• Lossy Image Compression with Conditional Diffusion Models [25.158390422252097]
  This paper outlines an end-to-end optimized lossy image compression framework using diffusion generative models. In contrast to VAE-based neural compression, where the (mean) decoder is a deterministic neural network, our decoder is a conditional diffusion model. Our approach yields stronger reported FID scores than the GAN-based model, while also yielding competitive performance with VAE-based models in several distortion metrics.
  arXiv  Detail & Related papers  (2022-09-14T21:53:27Z)
• Enhanced Invertible Encoding for Learned Image Compression [40.21904131503064]
  In this paper, we propose an enhanced Invertible. Network with invertible neural networks (INNs) to largely mitigate the information loss problem for better compression. Experimental results on the Kodak, CLIC, and Tecnick datasets show that our method outperforms the existing learned image compression methods.
  arXiv  Detail & Related papers  (2021-08-08T17:32:10Z)
• Modeling Lost Information in Lossy Image Compression [72.69327382643549]
  Lossy image compression is one of the most commonly used operators for digital images. We propose a novel invertible framework called Invertible Lossy Compression (ILC) to largely mitigate the information loss problem.
  arXiv  Detail & Related papers  (2020-06-22T04:04:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.