Editing Out-of-domain GAN Inversion via Differential Activations

• URL: http://arxiv.org/abs/2207.08134v1
• Date: Sun, 17 Jul 2022 10:34:58 GMT
• Title: Editing Out-of-domain GAN Inversion via Differential Activations
• Authors: Haorui Song, Yong Du, Tianyi Xiang, Junyu Dong, Jing Qin, Shengfeng He
• Abstract summary: We propose a novel GAN prior based editing framework to tackle the out-of-domain inversion problem with a composition-decomposition paradigm. With the aid of the generated Diff-CAM mask, a coarse reconstruction can intuitively be composited by the paired original and edited images. In the decomposition phase, we further present a GAN prior based deghosting network for separating the final fine edited image from the coarse reconstruction.
• Score: 56.62964029959131
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Despite the demonstrated editing capacity in the latent space of a pretrained GAN model, inverting real-world images is stuck in a dilemma that the reconstruction cannot be faithful to the original input. The main reason for this is that the distributions between training and real-world data are misaligned, and because of that, it is unstable of GAN inversion for real image editing. In this paper, we propose a novel GAN prior based editing framework to tackle the out-of-domain inversion problem with a composition-decomposition paradigm. In particular, during the phase of composition, we introduce a differential activation module for detecting semantic changes from a global perspective, \ie, the relative gap between the features of edited and unedited images. With the aid of the generated Diff-CAM mask, a coarse reconstruction can intuitively be composited by the paired original and edited images. In this way, the attribute-irrelevant regions can be survived in almost whole, while the quality of such an intermediate result is still limited by an unavoidable ghosting effect. Consequently, in the decomposition phase, we further present a GAN prior based deghosting network for separating the final fine edited image from the coarse reconstruction. Extensive experiments exhibit superiorities over the state-of-the-art methods, in terms of qualitative and quantitative evaluations. The robustness and flexibility of our method is also validated on both scenarios of single attribute and multi-attribute manipulations.

Related papers

• Denoising as Adaptation: Noise-Space Domain Adaptation for Image Restoration [64.84134880709625]
  We show that it is possible to perform domain adaptation via the noise space using diffusion models. In particular, by leveraging the unique property of how auxiliary conditional inputs influence the multi-step denoising process, we derive a meaningful diffusion loss. We present crucial strategies such as channel-shuffling layer and residual-swapping contrastive learning in the diffusion model.
  arXiv  Detail & Related papers  (2024-06-26T17:40:30Z)
• Tuning-Free Inversion-Enhanced Control for Consistent Image Editing [44.311286151669464]
  We present a novel approach called Tuning-free Inversion-enhanced Control (TIC) TIC correlates features from the inversion process with those from the sampling process to mitigate the inconsistency in DDIM reconstruction. We also propose a mask-guided attention concatenation strategy that combines contents from both the inversion and the naive DDIM editing processes.
  arXiv  Detail & Related papers  (2023-12-22T11:13:22Z)
• Spatial-Contextual Discrepancy Information Compensation for GAN Inversion [67.21442893265973]
  We introduce a novel spatial-contextual discrepancy information compensationbased GAN-inversion method (SDIC) SDIC bridges the gap in image details between the original image and the reconstructed/edited image. Our proposed method achieves the excellent distortion-editability trade-off at a fast inference speed for both image inversion and editing tasks.
  arXiv  Detail & Related papers  (2023-12-12T08:58:56Z)
• In-Domain GAN Inversion for Faithful Reconstruction and Editability [132.68255553099834]
  We propose in-domain GAN inversion, which consists of a domain-guided domain-regularized and a encoder to regularize the inverted code in the native latent space of the pre-trained GAN model. We make comprehensive analyses on the effects of the encoder structure, the starting inversion point, as well as the inversion parameter space, and observe the trade-off between the reconstruction quality and the editing property.
  arXiv  Detail & Related papers  (2023-09-25T08:42:06Z)
• RIGID: Recurrent GAN Inversion and Editing of Real Face Videos [73.97520691413006]
  GAN inversion is indispensable for applying the powerful editability of GAN to real images. Existing methods invert video frames individually often leading to undesired inconsistent results over time. We propose a unified recurrent framework, named textbfRecurrent vtextbfIdeo textbfGAN textbfInversion and etextbfDiting (RIGID) Our framework learns the inherent coherence between input frames in an end-to-end manner.
  arXiv  Detail & Related papers  (2023-08-11T12:17:24Z)
• In-N-Out: Faithful 3D GAN Inversion with Volumetric Decomposition for Face Editing [28.790900756506833]
  3D-aware GANs offer new capabilities for view synthesis while preserving the editing functionalities of their 2D counterparts. GAN inversion is a crucial step that seeks the latent code to reconstruct input images or videos, subsequently enabling diverse editing tasks through manipulation of this latent code. We address this issue by explicitly modeling OOD objects from the input in 3D-aware GANs.
  arXiv  Detail & Related papers  (2023-02-09T18:59:56Z)
• ReGANIE: Rectifying GAN Inversion Errors for Accurate Real Image Editing [20.39792009151017]
  StyleGAN allows for flexible and plausible editing of generated images by manipulating the semantic-rich latent style space. Projecting a real image into its latent space encounters an inherent trade-off between inversion quality and editability. We propose a novel two-phase framework by designating two separate networks to tackle editing and reconstruction respectively.
  arXiv  Detail & Related papers  (2023-01-31T04:38:42Z)
• High-Fidelity GAN Inversion for Image Attribute Editing [61.966946442222735]
  We present a novel high-fidelity generative adversarial network (GAN) inversion framework that enables attribute editing with image-specific details well-preserved. With a low bit-rate latent code, previous works have difficulties in preserving high-fidelity details in reconstructed and edited images. We propose a distortion consultation approach that employs a distortion map as a reference for high-fidelity reconstruction.
  arXiv  Detail & Related papers  (2021-09-14T11:23:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.