Iterative Facial Image Inpainting using Cyclic Reverse Generator
- URL: http://arxiv.org/abs/2101.07036v1
- Date: Mon, 18 Jan 2021 12:19:58 GMT
- Title: Iterative Facial Image Inpainting using Cyclic Reverse Generator
- Authors: Yahya Dogan and Hacer Yalim Keles
- Abstract summary: Cyclic Reverse Generator (CRG) architecture provides an encoder-generator model.
We empirically observed that only a few iterations are sufficient to generate realistic images with the proposed model.
Our method allows applying sketch-based inpaintings, using variety of mask types, and producing multiple and diverse results.
- Score: 0.913755431537592
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: Facial image inpainting is a challenging problem as it requires generating
new pixels that include semantic information for masked key components in a
face, e.g., eyes and nose. Recently, remarkable methods have been proposed in
this field. Most of these approaches use encoder-decoder architectures and have
different limitations such as allowing unique results for a given image and a
particular mask. Alternatively, some approaches generate promising results
using different masks with generator networks. However, these approaches are
optimization-based and usually require quite a number of iterations. In this
paper, we propose an efficient solution to the facial image painting problem
using the Cyclic Reverse Generator (CRG) architecture, which provides an
encoder-generator model. We use the encoder to embed a given image to the
generator space and incrementally inpaint the masked regions until a plausible
image is generated; a discriminator network is utilized to assess the generated
images during the iterations. We empirically observed that only a few
iterations are sufficient to generate realistic images with the proposed model.
After the generation process, for the post processing, we utilize a Unet model
that we trained specifically for this task to remedy the artifacts close to the
mask boundaries. Our method allows applying sketch-based inpaintings, using
variety of mask types, and producing multiple and diverse results. We
qualitatively compared our method with the state-of-the-art models and observed
that our method can compete with the other models in all mask types; it is
particularly better in images where larger masks are utilized.
Related papers
- Inpainting-Driven Mask Optimization for Object Removal [15.429649454099085]
This paper proposes a mask optimization method for improving the quality of object removal using image inpainting.
In our method, this domain gap is resolved by training the inpainting network with object masks extracted by segmentation.
To optimize the object masks for inpainting, the segmentation network is connected to the inpainting network and end-to-end trained to improve the inpainting performance.
arXiv Detail & Related papers (2024-03-23T13:52:16Z) - Improving Masked Autoencoders by Learning Where to Mask [65.89510231743692]
Masked image modeling is a promising self-supervised learning method for visual data.
We present AutoMAE, a framework that uses Gumbel-Softmax to interlink an adversarially-trained mask generator and a mask-guided image modeling process.
In our experiments, AutoMAE is shown to provide effective pretraining models on standard self-supervised benchmarks and downstream tasks.
arXiv Detail & Related papers (2023-03-12T05:28:55Z) - MaskSketch: Unpaired Structure-guided Masked Image Generation [56.88038469743742]
MaskSketch is an image generation method that allows spatial conditioning of the generation result using a guiding sketch as an extra conditioning signal during sampling.
We show that intermediate self-attention maps of a masked generative transformer encode important structural information of the input image.
Our results show that MaskSketch achieves high image realism and fidelity to the guiding structure.
arXiv Detail & Related papers (2023-02-10T20:27:02Z) - Shape-Aware Masking for Inpainting in Medical Imaging [49.61617087640379]
Inpainting has been proposed as a successful deep learning technique for unsupervised medical image model discovery.
We introduce a method for generating shape-aware masks for inpainting, which aims at learning the statistical shape prior.
We propose an unsupervised guided masking approach based on an off-the-shelf inpainting model and a superpixel over-segmentation algorithm.
arXiv Detail & Related papers (2022-07-12T18:35:17Z) - RePaint: Inpainting using Denoising Diffusion Probabilistic Models [161.74792336127345]
Free-form inpainting is the task of adding new content to an image in the regions specified by an arbitrary binary mask.
We propose RePaint: A Denoising Probabilistic Model (DDPM) based inpainting approach that is applicable to even extreme masks.
We validate our method for both faces and general-purpose image inpainting using standard and extreme masks.
arXiv Detail & Related papers (2022-01-24T18:40:15Z) - Learning Sparse Masks for Diffusion-based Image Inpainting [10.633099921979674]
Diffusion-based inpainting is a powerful tool for the reconstruction of images from sparse data.
We provide a model for highly efficient adaptive mask generation.
Experiments indicate that our model can achieve competitive quality with an acceleration by as much as four orders of magnitude.
arXiv Detail & Related papers (2021-10-06T10:20:59Z) - Few-shot Semantic Image Synthesis Using StyleGAN Prior [8.528384027684192]
We present a training strategy that performs pseudo labeling of semantic masks using the StyleGAN prior.
Our key idea is to construct a simple mapping between the StyleGAN feature and each semantic class from a few examples of semantic masks.
Although the pseudo semantic masks might be too coarse for previous approaches that require pixel-aligned masks, our framework can synthesize high-quality images from not only dense semantic masks but also sparse inputs such as landmarks and scribbles.
arXiv Detail & Related papers (2021-03-27T11:04:22Z) - Free-Form Image Inpainting via Contrastive Attention Network [64.05544199212831]
In image inpainting tasks, masks with any shapes can appear anywhere in images which form complex patterns.
It is difficult for encoders to capture such powerful representations under this complex situation.
We propose a self-supervised Siamese inference network to improve the robustness and generalization.
arXiv Detail & Related papers (2020-10-29T14:46:05Z) - R-MNet: A Perceptual Adversarial Network for Image Inpainting [5.471225956329675]
We propose a Wasserstein GAN combined with a new reverse mask operator, namely Reverse Masking Network (R-MNet), a perceptual adversarial network for image inpainting.
We show that our method is able to generalize to high-resolution inpainting task, and further show more realistic outputs that are plausible to the human visual system.
arXiv Detail & Related papers (2020-08-11T10:58:10Z) - Swapping Autoencoder for Deep Image Manipulation [94.33114146172606]
We propose the Swapping Autoencoder, a deep model designed specifically for image manipulation.
The key idea is to encode an image with two independent components and enforce that any swapped combination maps to a realistic image.
Experiments on multiple datasets show that our model produces better results and is substantially more efficient compared to recent generative models.
arXiv Detail & Related papers (2020-07-01T17:59:57Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.