High-fidelity GAN Inversion with Padding Space

• URL: http://arxiv.org/abs/2203.11105v1
• Date: Mon, 21 Mar 2022 16:32:12 GMT
• Title: High-fidelity GAN Inversion with Padding Space
• Authors: Qingyan Bai, Yinghao Xu, Jiapeng Zhu, Weihao Xia, Yujiu Yang, Yujun Shen
• Abstract summary: Inverting a Generative Adversarial Network (GAN) facilitates a wide range of image editing tasks using pre-trained generators. Existing methods typically employ the latent space of GANs as the inversion space yet observe the insufficient recovery of spatial details. We propose to involve the padding space of the generator to complement the latent space with spatial information.
• Score: 38.9258619444968
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Inverting a Generative Adversarial Network (GAN) facilitates a wide range of image editing tasks using pre-trained generators. Existing methods typically employ the latent space of GANs as the inversion space yet observe the insufficient recovery of spatial details. In this work, we propose to involve the padding space of the generator to complement the latent space with spatial information. Concretely, we replace the constant padding (e.g., usually zeros) used in convolution layers with some instance-aware coefficients. In this way, the inductive bias assumed in the pre-trained model can be appropriately adapted to fit each individual image. Through learning a carefully designed encoder, we manage to improve the inversion quality both qualitatively and quantitatively, outperforming existing alternatives. We then demonstrate that such a space extension barely affects the native GAN manifold, hence we can still reuse the prior knowledge learned by GANs for various downstream applications. Beyond the editing tasks explored in prior arts, our approach allows a more flexible image manipulation, such as the separate control of face contour and facial details, and enables a novel editing manner where users can customize their own manipulations highly efficiently.

Related papers

• A Compact and Semantic Latent Space for Disentangled and Controllable Image Editing [4.8201607588546]
  We propose an auto-encoder which re-organizes the latent space of StyleGAN, so that each attribute which we wish to edit corresponds to an axis of the new latent space. We show that our approach has greater disentanglement than competing methods, while maintaining fidelity to the original image with respect to identity.
  arXiv  Detail & Related papers  (2023-12-13T16:18:45Z)
• SC2GAN: Rethinking Entanglement by Self-correcting Correlated GAN Space [16.040942072859075]
  Gene Networks that achieve following editing directions for one attribute could result in entangled changes with other attributes. We propose a novel framework SC$2$GAN disentanglement by re-projecting low-density latent code samples in the original latent space.
  arXiv  Detail & Related papers  (2023-10-10T14:42:32Z)
• In-Domain GAN Inversion for Faithful Reconstruction and Editability [132.68255553099834]
  We propose in-domain GAN inversion, which consists of a domain-guided domain-regularized and a encoder to regularize the inverted code in the native latent space of the pre-trained GAN model. We make comprehensive analyses on the effects of the encoder structure, the starting inversion point, as well as the inversion parameter space, and observe the trade-off between the reconstruction quality and the editing property.
  arXiv  Detail & Related papers  (2023-09-25T08:42:06Z)
• GIFD: A Generative Gradient Inversion Method with Feature Domain Optimization [52.55628139825667]
  Federated Learning (FL) has emerged as a promising distributed machine learning framework to preserve clients' privacy. Recent studies find that an attacker can invert the shared gradients and recover sensitive data against an FL system by leveraging pre-trained generative adversarial networks (GAN) as prior knowledge. We propose textbfGradient textbfInversion over textbfFeature textbfDomains (GIFD), which disassembles the GAN model and searches the feature domains of the intermediate layers.
  arXiv  Detail & Related papers  (2023-08-09T04:34:21Z)
• Spatial Latent Representations in Generative Adversarial Networks for Image Generation [0.0]
  We define a family of spatial latent spaces for StyleGAN2. We show that our spaces are effective for image manipulation and encode semantic information well.
  arXiv  Detail & Related papers  (2023-03-25T20:01:11Z)
• Spatial Steerability of GANs via Self-Supervision from Discriminator [123.27117057804732]
  We propose a self-supervised approach to improve the spatial steerability of GANs without searching for steerable directions in the latent space. Specifically, we design randomly sampled Gaussian heatmaps to be encoded into the intermediate layers of generative models as spatial inductive bias. During inference, users can interact with the spatial heatmaps in an intuitive manner, enabling them to edit the output image by adjusting the scene layout, moving, or removing objects.
  arXiv  Detail & Related papers  (2023-01-20T07:36:29Z)
• Low-Rank Subspaces in GANs [101.48350547067628]
  This work introduces low-rank subspaces that enable more precise control of GAN generation. LowRankGAN is able to find the low-dimensional representation of attribute manifold. Experiments on state-of-the-art GAN models (including StyleGAN2 and BigGAN) trained on various datasets demonstrate the effectiveness of our LowRankGAN.
  arXiv  Detail & Related papers  (2021-06-08T16:16:32Z)
• Positional Encoding as Spatial Inductive Bias in GANs [97.6622154941448]
  SinGAN shows impressive capability in learning internal patch distribution despite its limited effective receptive field. In this work, we show that such capability, to a large extent, is brought by the implicit positional encoding when using zero padding in the generators. We propose a new multi-scale training strategy and demonstrate its effectiveness in the state-of-the-art unconditional generator StyleGAN2.
  arXiv  Detail & Related papers  (2020-12-09T18:27:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.