Exploiting Deep Generative Prior for Versatile Image Restoration and Manipulation

• URL: http://arxiv.org/abs/2003.13659v4
• Date: Mon, 20 Jul 2020 10:06:24 GMT
• Title: Exploiting Deep Generative Prior for Versatile Image Restoration and Manipulation
• Authors: Xingang Pan, Xiaohang Zhan, Bo Dai, Dahua Lin, Chen Change Loy, Ping Luo
• Abstract summary: This work presents an effective way to exploit the image prior captured by a generative adversarial network (GAN) trained on large-scale natural images. The deep generative prior (DGP) provides compelling results to restore missing semantics, e.g., color, patch, resolution, of various degraded images.
• Score: 181.08127307338654
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Learning a good image prior is a long-term goal for image restoration and manipulation. While existing methods like deep image prior (DIP) capture low-level image statistics, there are still gaps toward an image prior that captures rich image semantics including color, spatial coherence, textures, and high-level concepts. This work presents an effective way to exploit the image prior captured by a generative adversarial network (GAN) trained on large-scale natural images. As shown in Fig.1, the deep generative prior (DGP) provides compelling results to restore missing semantics, e.g., color, patch, resolution, of various degraded images. It also enables diverse image manipulation including random jittering, image morphing, and category transfer. Such highly flexible restoration and manipulation are made possible through relaxing the assumption of existing GAN-inversion methods, which tend to fix the generator. Notably, we allow the generator to be fine-tuned on-the-fly in a progressive manner regularized by feature distance obtained by the discriminator in GAN. We show that these easy-to-implement and practical changes help preserve the reconstruction to remain in the manifold of nature image, and thus lead to more precise and faithful reconstruction for real images. Code is available at https://github.com/XingangPan/deep-generative-prior.

Related papers

• Restoration by Generation with Constrained Priors [25.906981634736795]
  We propose a method to adapt a pretrained diffusion model for image restoration by simply adding noise to the input image to be restored and then denoise. We show superior performances on multiple real-world restoration datasets in preserving identity and image quality. This approach allows us to produce results that accurately preserve high-frequency details, which previous works are unable to do.
  arXiv  Detail & Related papers  (2023-12-28T17:50:54Z)
• In-Domain GAN Inversion for Faithful Reconstruction and Editability [132.68255553099834]
  We propose in-domain GAN inversion, which consists of a domain-guided domain-regularized and a encoder to regularize the inverted code in the native latent space of the pre-trained GAN model. We make comprehensive analyses on the effects of the encoder structure, the starting inversion point, as well as the inversion parameter space, and observe the trade-off between the reconstruction quality and the editing property.
  arXiv  Detail & Related papers  (2023-09-25T08:42:06Z)
• All-in-one Multi-degradation Image Restoration Network via Hierarchical Degradation Representation [47.00239809958627]
  We propose a novel All-in-one Multi-degradation Image Restoration Network (AMIRNet) AMIRNet learns a degradation representation for unknown degraded images by progressively constructing a tree structure through clustering. This tree-structured representation explicitly reflects the consistency and discrepancy of various distortions, providing a specific clue for image restoration.
  arXiv  Detail & Related papers  (2023-08-06T04:51:41Z)
• RGI: robust GAN-inversion for mask-free image inpainting and unsupervised pixel-wise anomaly detection [18.10039647382319]
  We propose a Robust GAN-inversion (RGI) method with a provable robustness guarantee to achieve image restoration under unknown textitgross corruptions. We show that the restored image and the identified corrupted region mask convergeally to the ground truth. The proposed RGI/R-RGI method unifies two important applications with state-of-the-art (SOTA) performance.
  arXiv  Detail & Related papers  (2023-02-24T05:43:03Z)
• High-Fidelity Image Inpainting with GAN Inversion [23.49170140410603]
  In this paper, we propose a novel GAN inversion model for image inpainting, dubbed InvertFill. Within the encoder, the pre-modulation network leverages multi-scale structures to encode more discriminative semantics into style vectors. To reconstruct faithful and photorealistic images, a simple yet effective Soft-update Mean Latent module is designed to capture more diverse in-domain patterns that synthesize high-fidelity textures for large corruptions.
  arXiv  Detail & Related papers  (2022-08-25T03:39:24Z)
• Restormer: Efficient Transformer for High-Resolution Image Restoration [118.9617735769827]
  convolutional neural networks (CNNs) perform well at learning generalizable image priors from large-scale data. Transformers have shown significant performance gains on natural language and high-level vision tasks. Our model, named Restoration Transformer (Restormer), achieves state-of-the-art results on several image restoration tasks.
  arXiv  Detail & Related papers  (2021-11-18T18:59:10Z)
• GLEAN: Generative Latent Bank for Large-Factor Image Super-Resolution [85.53811497840725]
  We show that Generative Adversarial Networks (GANs), e.g., StyleGAN, can be used as a latent bank to improve the restoration quality of large-factor image super-resolution (SR) Our method, Generative LatEnt bANk (GLEAN), goes beyond existing practices by directly leveraging rich and diverse priors encapsulated in a pre-trained GAN. Images upscaled by GLEAN show clear improvements in terms of fidelity and texture faithfulness in comparison to existing methods.
  arXiv  Detail & Related papers  (2020-12-01T18:56:14Z)
• The Power of Triply Complementary Priors for Image Compressive Sensing [89.14144796591685]
  We propose a joint low-rank deep (LRD) image model, which contains a pair of complementaryly trip priors. We then propose a novel hybrid plug-and-play framework based on the LRD model for image CS. To make the optimization tractable, a simple yet effective algorithm is proposed to solve the proposed H-based image CS problem.
  arXiv  Detail & Related papers  (2020-05-16T08:17:44Z)
• In-Domain GAN Inversion for Real Image Editing [56.924323432048304]
  A common practice of feeding a real image to a trained GAN generator is to invert it back to a latent code. Existing inversion methods typically focus on reconstructing the target image by pixel values yet fail to land the inverted code in the semantic domain of the original latent space. We propose an in-domain GAN inversion approach, which faithfully reconstructs the input image and ensures the inverted code to be semantically meaningful for editing.
  arXiv  Detail & Related papers  (2020-03-31T18:20:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.