Deep Generative Model for Image Inpainting with Local Binary Pattern Learning and Spatial Attention

• URL: http://arxiv.org/abs/2009.01031v1
• Date: Wed, 2 Sep 2020 12:59:28 GMT
• Title: Deep Generative Model for Image Inpainting with Local Binary Pattern Learning and Spatial Attention
• Authors: Haiwei Wu and Jiantao Zhou and Yuanman Li
• Abstract summary: We propose a new end-to-end, two-stage (coarse-to-fine) generative model through combining a local binary pattern (LBP) learning network with an actual inpainting network. Experiments on public datasets including CelebA-HQ, Places and Paris StreetView demonstrate that our model generates better inpainting results than the state-of-the-art competing algorithms.
• Score: 28.807711307545112
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep learning (DL) has demonstrated its powerful capabilities in the field of image inpainting. The DL-based image inpainting approaches can produce visually plausible results, but often generate various unpleasant artifacts, especially in the boundary and highly textured regions. To tackle this challenge, in this work, we propose a new end-to-end, two-stage (coarse-to-fine) generative model through combining a local binary pattern (LBP) learning network with an actual inpainting network. Specifically, the first LBP learning network using U-Net architecture is designed to accurately predict the structural information of the missing region, which subsequently guides the second image inpainting network for better filling the missing pixels. Furthermore, an improved spatial attention mechanism is integrated in the image inpainting network, by considering the consistency not only between the known region with the generated one, but also within the generated region itself. Extensive experiments on public datasets including CelebA-HQ, Places and Paris StreetView demonstrate that our model generates better inpainting results than the state-of-the-art competing algorithms, both quantitatively and qualitatively. The source code and trained models will be made available at https://github.com/HighwayWu/ImageInpainting.

Related papers

• Learning Prior Feature and Attention Enhanced Image Inpainting [63.21231753407192]
  This paper incorporates the pre-training based Masked AutoEncoder (MAE) into the inpainting model. We propose to use attention priors from MAE to make the inpainting model learn more long-distance dependencies between masked and unmasked regions.
  arXiv  Detail & Related papers  (2022-08-03T04:32:53Z)
• Contextual Attention Mechanism, SRGAN Based Inpainting System for Eliminating Interruptions from Images [2.894944733573589]
  We propose an end-to-end pipeline for inpainting images using a complete Machine Learning approach. We first use the YOLO model to automatically identify and localize the object we wish to remove from the image. After this, we provide the masked image and original image to the GAN model which uses the Contextual Attention method to fill in the region.
  arXiv  Detail & Related papers  (2022-04-06T05:51:04Z)
• Local and Global GANs with Semantic-Aware Upsampling for Image Generation [201.39323496042527]
  We consider generating images using local context. We propose a class-specific generative network using semantic maps as guidance. Lastly, we propose a novel semantic-aware upsampling method.
  arXiv  Detail & Related papers  (2022-02-28T19:24:25Z)
• Fully Context-Aware Image Inpainting with a Learned Semantic Pyramid [102.24539566851809]
  Restoring reasonable and realistic content for arbitrary missing regions in images is an important yet challenging task. Recent image inpainting models have made significant progress in generating vivid visual details, but they can still lead to texture blurring or structural distortions. We propose the Semantic Pyramid Network (SPN) motivated by the idea that learning multi-scale semantic priors can greatly benefit the recovery of locally missing content in images.
  arXiv  Detail & Related papers  (2021-12-08T04:33:33Z)
• Boosting Image Outpainting with Semantic Layout Prediction [18.819765707811904]
  We train a GAN to extend regions in semantic segmentation domain instead of image domain. Another GAN model is trained to synthesize real images based on the extended semantic layouts. Our approach can handle semantic clues more easily and hence works better in complex scenarios.
  arXiv  Detail & Related papers  (2021-10-18T13:09:31Z)
• Generating Diverse Structure for Image Inpainting With Hierarchical VQ-VAE [74.29384873537587]
  We propose a two-stage model for diverse inpainting, where the first stage generates multiple coarse results each of which has a different structure, and the second stage refines each coarse result separately by augmenting texture. Experimental results on CelebA-HQ, Places2, and ImageNet datasets show that our method not only enhances the diversity of the inpainting solutions but also improves the visual quality of the generated multiple images.
  arXiv  Detail & Related papers  (2021-03-18T05:10:49Z)
• Free-Form Image Inpainting via Contrastive Attention Network [64.05544199212831]
  In image inpainting tasks, masks with any shapes can appear anywhere in images which form complex patterns. It is difficult for encoders to capture such powerful representations under this complex situation. We propose a self-supervised Siamese inference network to improve the robustness and generalization.
  arXiv  Detail & Related papers  (2020-10-29T14:46:05Z)
• Enhanced Residual Networks for Context-based Image Outpainting [0.0]
  Deep models struggle to understand context and extrapolation through retained information. Current models use generative adversarial networks to generate results which lack localized image feature consistency and appear fake. We propose two methods to improve this issue: the use of a local and global discriminator, and the addition of residual blocks within the encoding section of the network.
  arXiv  Detail & Related papers  (2020-05-14T05:14:26Z)
• Very Long Natural Scenery Image Prediction by Outpainting [96.8509015981031]
  Outpainting receives less attention due to two challenges in it. First challenge is how to keep the spatial and content consistency between generated images and original input. Second challenge is how to maintain high quality in generated results.
  arXiv  Detail & Related papers  (2019-12-29T16:29:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.