Robust GAN inversion

• URL: http://arxiv.org/abs/2308.16510v1
• Date: Thu, 31 Aug 2023 07:47:11 GMT
• Title: Robust GAN inversion
• Authors: Egor Sevriugov, Ivan Oseledets
• Abstract summary: We propose an approach which works in native latent space $W$ and tunes the generator network to restore missing image details. We demonstrate the effectiveness of our approach on two complex datasets: Flickr-Faces-HQ and LSUN Church.
• Score: 5.1359892878090845
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advancements in real image editing have been attributed to the exploration of Generative Adversarial Networks (GANs) latent space. However, the main challenge of this procedure is GAN inversion, which aims to map the image to the latent space accurately. Existing methods that work on extended latent space $W+$ are unable to achieve low distortion and high editability simultaneously. To address this issue, we propose an approach which works in native latent space $W$ and tunes the generator network to restore missing image details. We introduce a novel regularization strategy with learnable coefficients obtained by training randomized StyleGAN 2 model - WRanGAN. This method outperforms traditional approaches in terms of reconstruction quality and computational efficiency, achieving the lowest distortion with 4 times fewer parameters. Furthermore, we observe a slight improvement in the quality of constructing hyperplanes corresponding to binary image attributes. We demonstrate the effectiveness of our approach on two complex datasets: Flickr-Faces-HQ and LSUN Church.

Related papers

• Realistic Extreme Image Rescaling via Generative Latent Space Learning [51.85790402171696]
  We propose a novel framework called Latent Space Based Image Rescaling (LSBIR) for extreme image rescaling tasks. LSBIR effectively leverages powerful natural image priors learned by a pre-trained text-to-image diffusion model to generate realistic HR images. In the first stage, a pseudo-invertible encoder-decoder models the bidirectional mapping between the latent features of the HR image and the target-sized LR image. In the second stage, the reconstructed features from the first stage are refined by a pre-trained diffusion model to generate more faithful and visually pleasing details.
  arXiv  Detail & Related papers  (2024-08-17T09:51:42Z)
• Spatial-Contextual Discrepancy Information Compensation for GAN Inversion [67.21442893265973]
  We introduce a novel spatial-contextual discrepancy information compensationbased GAN-inversion method (SDIC) SDIC bridges the gap in image details between the original image and the reconstructed/edited image. Our proposed method achieves the excellent distortion-editability trade-off at a fast inference speed for both image inversion and editing tasks.
  arXiv  Detail & Related papers  (2023-12-12T08:58:56Z)
• Meta-Auxiliary Network for 3D GAN Inversion [18.777352198191004]
  In this work, we present a novel meta-auxiliary framework, while leveraging the newly developed 3D GANs as generator. In the first stage, we invert the input image to an editable latent code using off-the-shelf inversion techniques. The auxiliary network is proposed to refine the generator parameters with the given image as input, which both predicts offsets for weights of convolutional layers and sampling positions of volume rendering. In the second stage, we perform meta-learning to fast adapt the auxiliary network to the input image, then the final reconstructed image is synthesized via the meta-learned auxiliary network.
  arXiv  Detail & Related papers  (2023-05-18T11:26:27Z)
• ReGANIE: Rectifying GAN Inversion Errors for Accurate Real Image Editing [20.39792009151017]
  StyleGAN allows for flexible and plausible editing of generated images by manipulating the semantic-rich latent style space. Projecting a real image into its latent space encounters an inherent trade-off between inversion quality and editability. We propose a novel two-phase framework by designating two separate networks to tackle editing and reconstruction respectively.
  arXiv  Detail & Related papers  (2023-01-31T04:38:42Z)
• Overparameterization Improves StyleGAN Inversion [66.8300251627992]
  Existing inversion approaches obtain promising yet imperfect results. We show that this allows us to obtain near-perfect image reconstruction without the need for encoders. Our approach also retains editability, which we demonstrate by realistically interpolating between images.
  arXiv  Detail & Related papers  (2022-05-12T18:42:43Z)
• One-Shot Adaptation of GAN in Just One CLIP [51.188396199083336]
  We present a novel single-shot GAN adaptation method through unified CLIP space manipulations. Specifically, our model employs a two-step training strategy: reference image search in the source generator using a CLIP-guided latent optimization. We show that our model generates diverse outputs with the target texture and outperforms the baseline models both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2022-03-17T13:03:06Z)
• HyperInverter: Improving StyleGAN Inversion via Hypernetwork [12.173568611144628]
  Current GAN inversion methods fail to meet at least one of the three requirements listed below: high reconstruction quality, editability, and fast inference. We present a novel two-phase strategy in this research that fits all requirements at the same time. Our method is entirely encoder-based, resulting in extremely fast inference.
  arXiv  Detail & Related papers  (2021-12-01T18:56:05Z)
• Spatial-Separated Curve Rendering Network for Efficient and High-Resolution Image Harmonization [59.19214040221055]
  We propose a novel spatial-separated curve rendering network (S$2$CRNet) for efficient and high-resolution image harmonization. The proposed method reduces more than 90% parameters compared with previous methods. Our method can work smoothly on higher resolution images in real-time which is more than 10$times$ faster than the existing methods.
  arXiv  Detail & Related papers  (2021-09-13T07:20:16Z)
• Efficient texture-aware multi-GAN for image inpainting [5.33024001730262]
  Recent GAN-based (Generative adversarial networks) inpainting methods show remarkable improvements. We propose a multi-GAN architecture improving both the performance and rendering efficiency.
  arXiv  Detail & Related papers  (2020-09-30T14:58:03Z)
• Image Fine-grained Inpainting [89.17316318927621]
  We present a one-stage model that utilizes dense combinations of dilated convolutions to obtain larger and more effective receptive fields. To better train this efficient generator, except for frequently-used VGG feature matching loss, we design a novel self-guided regression loss. We also employ a discriminator with local and global branches to ensure local-global contents consistency.
  arXiv  Detail & Related papers  (2020-02-07T03:45:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.