Realistic Extreme Image Rescaling via Generative Latent Space Learning

• URL: http://arxiv.org/abs/2408.09151v1
• Date: Sat, 17 Aug 2024 09:51:42 GMT
• Title: Realistic Extreme Image Rescaling via Generative Latent Space Learning
• Authors: Ce Wang, Wanjie Sun, Zhenzhong Chen,
• Abstract summary: We propose a novel framework called Latent Space Based Image Rescaling (LSBIR) for extreme image rescaling tasks. LSBIR effectively leverages powerful natural image priors learned by a pre-trained text-to-image diffusion model to generate realistic HR images. In the first stage, a pseudo-invertible encoder-decoder models the bidirectional mapping between the latent features of the HR image and the target-sized LR image. In the second stage, the reconstructed features from the first stage are refined by a pre-trained diffusion model to generate more faithful and visually pleasing details.
• Score: 51.85790402171696
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Image rescaling aims to learn the optimal downscaled low-resolution (LR) image that can be accurately reconstructed to its original high-resolution (HR) counterpart. This process is crucial for efficient image processing and storage, especially in the era of ultra-high definition media. However, extreme downscaling factors pose significant challenges due to the highly ill-posed nature of the inverse upscaling process, causing existing methods to struggle in generating semantically plausible structures and perceptually rich textures. In this work, we propose a novel framework called Latent Space Based Image Rescaling (LSBIR) for extreme image rescaling tasks. LSBIR effectively leverages powerful natural image priors learned by a pre-trained text-to-image diffusion model to generate realistic HR images. The rescaling is performed in the latent space of a pre-trained image encoder and decoder, which offers better perceptual reconstruction quality due to its stronger sparsity and richer semantics. LSBIR adopts a two-stage training strategy. In the first stage, a pseudo-invertible encoder-decoder models the bidirectional mapping between the latent features of the HR image and the target-sized LR image. In the second stage, the reconstructed features from the first stage are refined by a pre-trained diffusion model to generate more faithful and visually pleasing details. Extensive experiments demonstrate the superiority of LSBIR over previous methods in both quantitative and qualitative evaluations. The code will be available at: https://github.com/wwangcece/LSBIR.

Related papers

• Harnessing Diffusion-Yielded Score Priors for Image Restoration [29.788482710572307]
  Deep image restoration models aim to learn a mapping from degraded image space to natural image space.<n>Three major classes of methods have emerged, including MSE-based, GAN-based, and diffusion-based methods.<n>We propose a novel method, HYPIR, to address these challenges.
  arXiv  Detail & Related papers  (2025-07-28T07:55:34Z)
• Effective Diffusion Transformer Architecture for Image Super-Resolution [63.254644431016345]
  We design an effective diffusion transformer for image super-resolution (DiT-SR) In practice, DiT-SR leverages an overall U-shaped architecture, and adopts a uniform isotropic design for all the transformer blocks. We analyze the limitation of the widely used AdaLN, and present a frequency-adaptive time-step conditioning module.
  arXiv  Detail & Related papers  (2024-09-29T07:14:16Z)
• Diff-Restorer: Unleashing Visual Prompts for Diffusion-based Universal Image Restoration [19.87693298262894]
  We propose Diff-Restorer, a universal image restoration method based on the diffusion model. We utilize the pre-trained visual language model to extract visual prompts from degraded images. We also design a Degradation-aware Decoder to perform structural correction and convert the latent code to the pixel domain.
  arXiv  Detail & Related papers  (2024-07-04T05:01:10Z)
• SSP-IR: Semantic and Structure Priors for Diffusion-based Realistic Image Restoration [20.873676111265656]
  SSP-IR aims to fully exploit semantic and structure priors from low-quality images. Our method outperforms other state-of-the-art methods overall on both synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2024-07-04T04:55:14Z)
• CasSR: Activating Image Power for Real-World Image Super-Resolution [24.152495730507823]
  Cascaded diffusion for Super-Resolution, CasSR, is a novel method designed to produce highly detailed and realistic images. We develop a cascaded controllable diffusion model that aims to optimize the extraction of information from low-resolution images.
  arXiv  Detail & Related papers  (2024-03-18T03:59:43Z)
• Mitigating Data Consistency Induced Discrepancy in Cascaded Diffusion Models for Sparse-view CT Reconstruction [4.227116189483428]
  This study introduces a novel Cascaded Diffusion with Discrepancy Mitigation framework. It includes the low-quality image generation in latent space and the high-quality image generation in pixel space. It minimizes computational costs by moving some inference steps from pixel space to latent space.
  arXiv  Detail & Related papers  (2024-03-14T12:58:28Z)
• JoReS-Diff: Joint Retinex and Semantic Priors in Diffusion Model for Low-light Image Enhancement [69.6035373784027]
  Low-light image enhancement (LLIE) has achieved promising performance by employing conditional diffusion models. Previous methods may neglect the importance of a sufficient formulation of task-specific condition strategy. We propose JoReS-Diff, a novel approach that incorporates Retinex- and semantic-based priors as the additional pre-processing condition.
  arXiv  Detail & Related papers  (2023-12-20T08:05:57Z)
• Steerable Conditional Diffusion for Out-of-Distribution Adaptation in Medical Image Reconstruction [75.91471250967703]
  We introduce a novel sampling framework called Steerable Conditional Diffusion. This framework adapts the diffusion model, concurrently with image reconstruction, based solely on the information provided by the available measurement. We achieve substantial enhancements in out-of-distribution performance across diverse imaging modalities.
  arXiv  Detail & Related papers  (2023-08-28T08:47:06Z)
• Low-Light Image Enhancement with Wavelet-based Diffusion Models [50.632343822790006]
  Diffusion models have achieved promising results in image restoration tasks, yet suffer from time-consuming, excessive computational resource consumption, and unstable restoration. We propose a robust and efficient Diffusion-based Low-Light image enhancement approach, dubbed DiffLL.
  arXiv  Detail & Related papers  (2023-06-01T03:08:28Z)
• A Unified Conditional Framework for Diffusion-based Image Restoration [39.418415473235235]
  We present a unified conditional framework based on diffusion models for image restoration. We leverage a lightweight UNet to predict initial guidance and the diffusion model to learn the residual of the guidance. To handle high-resolution images, we propose a simple yet effective inter-step patch-splitting strategy.
  arXiv  Detail & Related papers  (2023-05-31T17:22:24Z)
• Hierarchical Integration Diffusion Model for Realistic Image Deblurring [71.76410266003917]
  Diffusion models (DMs) have been introduced in image deblurring and exhibited promising performance. We propose the Hierarchical Integration Diffusion Model (HI-Diff), for realistic image deblurring. Experiments on synthetic and real-world blur datasets demonstrate that our HI-Diff outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-05-22T12:18:20Z)
• Super-resolution Reconstruction of Single Image for Latent features [8.857209365343646]
  Single-image super-resolution (SISR) typically focuses on restoring various degraded low-resolution (LR) images to a single high-resolution (HR) image. It is often challenging for models to simultaneously maintain high quality and rapid sampling while preserving diversity in details and texture features. This challenge can lead to issues such as model collapse, lack of rich details and texture features in the reconstructed HR images, and excessive time consumption for model sampling.
  arXiv  Detail & Related papers  (2022-11-16T09:37:07Z)
• Invertible Image Rescaling [118.2653765756915]
  We develop an Invertible Rescaling Net (IRN) to produce visually-pleasing low-resolution images. We capture the distribution of the lost information using a latent variable following a specified distribution in the downscaling process.
  arXiv  Detail & Related papers  (2020-05-12T09:55:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.