Spatial-Contextual Discrepancy Information Compensation for GAN Inversion

• URL: http://arxiv.org/abs/2312.07079v1
• Date: Tue, 12 Dec 2023 08:58:56 GMT
• Title: Spatial-Contextual Discrepancy Information Compensation for GAN Inversion
• Authors: Ziqiang Zhang, Yan Yan, Jing-Hao Xue, Hanzi Wang
• Abstract summary: We introduce a novel spatial-contextual discrepancy information compensationbased GAN-inversion method (SDIC) SDIC bridges the gap in image details between the original image and the reconstructed/edited image. Our proposed method achieves the excellent distortion-editability trade-off at a fast inference speed for both image inversion and editing tasks.
• Score: 67.21442893265973
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Most existing GAN inversion methods either achieve accurate reconstruction but lack editability or offer strong editability at the cost of fidelity. Hence, how to balance the distortioneditability trade-off is a significant challenge for GAN inversion. To address this challenge, we introduce a novel spatial-contextual discrepancy information compensationbased GAN-inversion method (SDIC), which consists of a discrepancy information prediction network (DIPN) and a discrepancy information compensation network (DICN). SDIC follows a "compensate-and-edit" paradigm and successfully bridges the gap in image details between the original image and the reconstructed/edited image. On the one hand, DIPN encodes the multi-level spatial-contextual information of the original and initial reconstructed images and then predicts a spatial-contextual guided discrepancy map with two hourglass modules. In this way, a reliable discrepancy map that models the contextual relationship and captures finegrained image details is learned. On the other hand, DICN incorporates the predicted discrepancy information into both the latent code and the GAN generator with different transformations, generating high-quality reconstructed/edited images. This effectively compensates for the loss of image details during GAN inversion. Both quantitative and qualitative experiments demonstrate that our proposed method achieves the excellent distortion-editability trade-off at a fast inference speed for both image inversion and editing tasks.

Related papers

• Gradual Residuals Alignment: A Dual-Stream Framework for GAN Inversion and Image Attribute Editing [36.01737879983636]
  GAN-based image editing firstly leverages GAN Inversion to project real images into the latent space of GAN and then manipulates corresponding latent codes. Recent inversion methods mainly utilize additional high-bit features to improve image details preservation. During editing, existing works fail to accurately complement the lost details and suffer from poor editability.
  arXiv  Detail & Related papers  (2024-02-22T09:28:47Z)
• In-Domain GAN Inversion for Faithful Reconstruction and Editability [132.68255553099834]
  We propose in-domain GAN inversion, which consists of a domain-guided domain-regularized and a encoder to regularize the inverted code in the native latent space of the pre-trained GAN model. We make comprehensive analyses on the effects of the encoder structure, the starting inversion point, as well as the inversion parameter space, and observe the trade-off between the reconstruction quality and the editing property.
  arXiv  Detail & Related papers  (2023-09-25T08:42:06Z)
• ReGANIE: Rectifying GAN Inversion Errors for Accurate Real Image Editing [20.39792009151017]
  StyleGAN allows for flexible and plausible editing of generated images by manipulating the semantic-rich latent style space. Projecting a real image into its latent space encounters an inherent trade-off between inversion quality and editability. We propose a novel two-phase framework by designating two separate networks to tackle editing and reconstruction respectively.
  arXiv  Detail & Related papers  (2023-01-31T04:38:42Z)
• PC-GANs: Progressive Compensation Generative Adversarial Networks for Pan-sharpening [50.943080184828524]
  We propose a novel two-step model for pan-sharpening that sharpens the MS image through the progressive compensation of the spatial and spectral information. The whole model is composed of triple GANs, and based on the specific architecture, a joint compensation loss function is designed to enable the triple GANs to be trained simultaneously.
  arXiv  Detail & Related papers  (2022-07-29T03:09:21Z)
• Editing Out-of-domain GAN Inversion via Differential Activations [56.62964029959131]
  We propose a novel GAN prior based editing framework to tackle the out-of-domain inversion problem with a composition-decomposition paradigm. With the aid of the generated Diff-CAM mask, a coarse reconstruction can intuitively be composited by the paired original and edited images. In the decomposition phase, we further present a GAN prior based deghosting network for separating the final fine edited image from the coarse reconstruction.
  arXiv  Detail & Related papers  (2022-07-17T10:34:58Z)
• Dual Perceptual Loss for Single Image Super-Resolution Using ESRGAN [13.335546116599494]
  This paper proposes a method called Dual Perceptual Loss (DP Loss) to replace the original perceptual loss to solve the problem of single image super-resolution reconstruction. Due to the complementary property between the VGG features and the ResNet features, the proposed DP Loss considers the advantages of learning two features simultaneously. The qualitative and quantitative analysis on benchmark datasets demonstrates the superiority of our proposed method over state-of-the-art super-resolution methods.
  arXiv  Detail & Related papers  (2022-01-17T12:42:56Z)
• High-Fidelity GAN Inversion for Image Attribute Editing [61.966946442222735]
  We present a novel high-fidelity generative adversarial network (GAN) inversion framework that enables attribute editing with image-specific details well-preserved. With a low bit-rate latent code, previous works have difficulties in preserving high-fidelity details in reconstructed and edited images. We propose a distortion consultation approach that employs a distortion map as a reference for high-fidelity reconstruction.
  arXiv  Detail & Related papers  (2021-09-14T11:23:48Z)
• Spatially-Adaptive Image Restoration using Distortion-Guided Networks [51.89245800461537]
  We present a learning-based solution for restoring images suffering from spatially-varying degradations. We propose SPAIR, a network design that harnesses distortion-localization information and dynamically adjusts to difficult regions in the image.
  arXiv  Detail & Related papers  (2021-08-19T11:02:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.