Diverse Inpainting and Editing with GAN Inversion

• URL: http://arxiv.org/abs/2307.15033v1
• Date: Thu, 27 Jul 2023 17:41:36 GMT
• Title: Diverse Inpainting and Editing with GAN Inversion
• Authors: Ahmet Burak Yildirim, Hamza Pehlivan, Bahri Batuhan Bilecen, Aysegul Dundar
• Abstract summary: Recent inversion methods have shown that real images can be inverted into StyleGAN's latent space. In this paper, we tackle an even more difficult task, inverting erased images into GAN's latent space for realistic inpaintings and editings.
• Score: 4.234367850767171
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent inversion methods have shown that real images can be inverted into StyleGAN's latent space and numerous edits can be achieved on those images thanks to the semantically rich feature representations of well-trained GAN models. However, extensive research has also shown that image inversion is challenging due to the trade-off between high-fidelity reconstruction and editability. In this paper, we tackle an even more difficult task, inverting erased images into GAN's latent space for realistic inpaintings and editings. Furthermore, by augmenting inverted latent codes with different latent samples, we achieve diverse inpaintings. Specifically, we propose to learn an encoder and mixing network to combine encoded features from erased images with StyleGAN's mapped features from random samples. To encourage the mixing network to utilize both inputs, we train the networks with generated data via a novel set-up. We also utilize higher-rate features to prevent color inconsistencies between the inpainted and unerased parts. We run extensive experiments and compare our method with state-of-the-art inversion and inpainting methods. Qualitative metrics and visual comparisons show significant improvements.

Related papers

• Warping the Residuals for Image Editing with StyleGAN [5.733811543584874]
  StyleGAN models show editing capabilities via their semantically interpretable latent organizations. Many works have been proposed for inverting images into StyleGAN's latent space. We present a novel image inversion architecture that extracts high-rate latent features and includes a flow estimation module.
  arXiv  Detail & Related papers  (2023-12-18T18:24:18Z)
• In-Domain GAN Inversion for Faithful Reconstruction and Editability [132.68255553099834]
  We propose in-domain GAN inversion, which consists of a domain-guided domain-regularized and a encoder to regularize the inverted code in the native latent space of the pre-trained GAN model. We make comprehensive analyses on the effects of the encoder structure, the starting inversion point, as well as the inversion parameter space, and observe the trade-off between the reconstruction quality and the editing property.
  arXiv  Detail & Related papers  (2023-09-25T08:42:06Z)
• StyleRes: Transforming the Residuals for Real Image Editing with StyleGAN [4.7590051176368915]
  Inverting real images into StyleGAN's latent space is an extensively studied problem. Trade-off between the image reconstruction fidelity and image editing quality remains an open challenge. We present a novel image inversion framework and a training pipeline to achieve high-fidelity image inversion with high-quality editing.
  arXiv  Detail & Related papers  (2022-12-29T16:14:09Z)
• High-Fidelity Image Inpainting with GAN Inversion [23.49170140410603]
  In this paper, we propose a novel GAN inversion model for image inpainting, dubbed InvertFill. Within the encoder, the pre-modulation network leverages multi-scale structures to encode more discriminative semantics into style vectors. To reconstruct faithful and photorealistic images, a simple yet effective Soft-update Mean Latent module is designed to capture more diverse in-domain patterns that synthesize high-fidelity textures for large corruptions.
  arXiv  Detail & Related papers  (2022-08-25T03:39:24Z)
• V-LinkNet: Learning Contextual Inpainting Across Latent Space of Generative Adversarial Network [7.5089719291325325]
  We propose the V-LinkNet cross-space learning strategy network to improve learning on contextualised features. We compare inpainting performance on the same face with different masks and on different faces with the same masks. Our result surpasses the state of the art when evaluated on the CelebA-HQ with the standard protocol.
  arXiv  Detail & Related papers  (2022-01-02T09:14:23Z)
• Ensembling with Deep Generative Views [72.70801582346344]
  generative models can synthesize "views" of artificial images that mimic real-world variations, such as changes in color or pose. Here, we investigate whether such views can be applied to real images to benefit downstream analysis tasks such as image classification. We use StyleGAN2 as the source of generative augmentations and investigate this setup on classification tasks involving facial attributes, cat faces, and cars.
  arXiv  Detail & Related papers  (2021-04-29T17:58:35Z)
• In&Out : Diverse Image Outpainting via GAN Inversion [89.84841983778672]
  Image outpainting seeks for a semantically consistent extension of the input image beyond its available content. In this work, we formulate the problem from the perspective of inverting generative adversarial networks. Our generator renders micro-patches conditioned on their joint latent code as well as their individual positions in the image.
  arXiv  Detail & Related papers  (2021-04-01T17:59:10Z)
• Designing an Encoder for StyleGAN Image Manipulation [38.909059126878354]
  We study the latent space of StyleGAN, the state-of-the-art unconditional generator. We identify and analyze the existence of a distortion-editability tradeoff and a distortion-perception tradeoff within the StyleGAN latent space. We present an encoder based on our two principles that is specifically designed for facilitating editing on real images.
  arXiv  Detail & Related papers  (2021-02-04T17:52:38Z)
• Free-Form Image Inpainting via Contrastive Attention Network [64.05544199212831]
  In image inpainting tasks, masks with any shapes can appear anywhere in images which form complex patterns. It is difficult for encoders to capture such powerful representations under this complex situation. We propose a self-supervised Siamese inference network to improve the robustness and generalization.
  arXiv  Detail & Related papers  (2020-10-29T14:46:05Z)
• Swapping Autoencoder for Deep Image Manipulation [94.33114146172606]
  We propose the Swapping Autoencoder, a deep model designed specifically for image manipulation. The key idea is to encode an image with two independent components and enforce that any swapped combination maps to a realistic image. Experiments on multiple datasets show that our model produces better results and is substantially more efficient compared to recent generative models.
  arXiv  Detail & Related papers  (2020-07-01T17:59:57Z)
• In-Domain GAN Inversion for Real Image Editing [56.924323432048304]
  A common practice of feeding a real image to a trained GAN generator is to invert it back to a latent code. Existing inversion methods typically focus on reconstructing the target image by pixel values yet fail to land the inverted code in the semantic domain of the original latent space. We propose an in-domain GAN inversion approach, which faithfully reconstructs the input image and ensures the inverted code to be semantically meaningful for editing.
  arXiv  Detail & Related papers  (2020-03-31T18:20:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.