JPGNet: Joint Predictive Filtering and Generative Network for Image Inpainting

• URL: http://arxiv.org/abs/2107.04281v1
• Date: Fri, 9 Jul 2021 07:49:52 GMT
• Title: JPGNet: Joint Predictive Filtering and Generative Network for Image Inpainting
• Authors: Xiaoguang Li and Qing Guo and Felix Juefei-Xu and Hongkai Yu and Yang Liu and Song wang
• Abstract summary: Image inpainting aims to restore the missing regions and make the recovery results identical to the originally complete image. Existing works usually regard it as a pure generation problem and employ cutting-edge generative techniques to address it. In this paper, we formulate image inpainting as a mix of two problems, i.e., predictive filtering and deep generation.
• Score: 21.936689731138213
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Image inpainting aims to restore the missing regions and make the recovery results identical to the originally complete image, which is different from the common generative task emphasizing the naturalness of generated images. Nevertheless, existing works usually regard it as a pure generation problem and employ cutting-edge generative techniques to address it. The generative networks fill the main missing parts with realistic contents but usually distort the local structures. In this paper, we formulate image inpainting as a mix of two problems, i.e., predictive filtering and deep generation. Predictive filtering is good at preserving local structures and removing artifacts but falls short to complete the large missing regions. The deep generative network can fill the numerous missing pixels based on the understanding of the whole scene but hardly restores the details identical to the original ones. To make use of their respective advantages, we propose the joint predictive filtering and generative network (JPGNet) that contains three branches: predictive filtering & uncertainty network (PFUNet), deep generative network, and uncertainty-aware fusion network (UAFNet). The PFUNet can adaptively predict pixel-wise kernels for filtering-based inpainting according to the input image and output an uncertainty map. This map indicates the pixels should be processed by filtering or generative networks, which is further fed to the UAFNet for a smart combination between filtering and generative results. Note that, our method as a novel framework for the image inpainting problem can benefit any existing generation-based methods. We validate our method on three public datasets, i.e., Dunhuang, Places2, and CelebA, and demonstrate that our method can enhance three state-of-the-art generative methods (i.e., StructFlow, EdgeConnect, and RFRNet) significantly with the slightly extra time cost.

Related papers

• Multi-Prior Learning via Neural Architecture Search for Blind Face Restoration [61.27907052910136]
  Blind Face Restoration (BFR) aims to recover high-quality face images from low-quality ones. Current methods still suffer from two major difficulties: 1) how to derive a powerful network architecture without extensive hand tuning; 2) how to capture complementary information from multiple facial priors in one network to improve restoration performance. We propose a Face Restoration Searching Network (FRSNet) to adaptively search the suitable feature extraction architecture within our specified search space.
  arXiv  Detail & Related papers  (2022-06-28T12:29:53Z)
• Contextual Attention Mechanism, SRGAN Based Inpainting System for Eliminating Interruptions from Images [2.894944733573589]
  We propose an end-to-end pipeline for inpainting images using a complete Machine Learning approach. We first use the YOLO model to automatically identify and localize the object we wish to remove from the image. After this, we provide the masked image and original image to the GAN model which uses the Contextual Attention method to fill in the region.
  arXiv  Detail & Related papers  (2022-04-06T05:51:04Z)
• Location-Free Camouflage Generation Network [82.74353843283407]
  Camouflage is a common visual phenomenon, which refers to hiding the foreground objects into the background images, making them briefly invisible to the human eye. This paper proposes a novel Location-free Camouflage Generation Network (LCG-Net) that fuse high-level features of foreground and background image, and generate result by one inference. Experiments show that our method has results as satisfactory as state-of-the-art in the single-appearance regions and are less likely to be completely invisible, but far exceed the quality of the state-of-the-art in the multi-appearance regions.
  arXiv  Detail & Related papers  (2022-03-18T10:33:40Z)
• MISF: Multi-level Interactive Siamese Filtering for High-Fidelity Image Inpainting [35.79101039727397]
  We study the advantages and challenges of image-level predictive filtering for image inpainting. We propose a novel filtering technique, i.e., Multilevel Interactive Siamese Filtering (MISF), which contains two branches: kernel prediction branch (KPB) and semantic & image filtering branch (SIFB) Our method outperforms state-of-the-art baselines on four metrics, i.e., L1, PSNR, SSIM, and LPIPS.
  arXiv  Detail & Related papers  (2022-03-12T01:32:39Z)
• Free-Form Image Inpainting via Contrastive Attention Network [64.05544199212831]
  In image inpainting tasks, masks with any shapes can appear anywhere in images which form complex patterns. It is difficult for encoders to capture such powerful representations under this complex situation. We propose a self-supervised Siamese inference network to improve the robustness and generalization.
  arXiv  Detail & Related papers  (2020-10-29T14:46:05Z)
• R-MNet: A Perceptual Adversarial Network for Image Inpainting [5.471225956329675]
  We propose a Wasserstein GAN combined with a new reverse mask operator, namely Reverse Masking Network (R-MNet), a perceptual adversarial network for image inpainting. We show that our method is able to generalize to high-resolution inpainting task, and further show more realistic outputs that are plausible to the human visual system.
  arXiv  Detail & Related papers  (2020-08-11T10:58:10Z)
• Image Fine-grained Inpainting [89.17316318927621]
  We present a one-stage model that utilizes dense combinations of dilated convolutions to obtain larger and more effective receptive fields. To better train this efficient generator, except for frequently-used VGG feature matching loss, we design a novel self-guided regression loss. We also employ a discriminator with local and global branches to ensure local-global contents consistency.
  arXiv  Detail & Related papers  (2020-02-07T03:45:25Z)
• Recognizing Instagram Filtered Images with Feature De-stylization [81.38905784617089]
  This paper presents a study on how popular pretrained models are affected by commonly used Instagram filters. Our analysis suggests that simple structure preserving filters which only alter the global appearance of an image can lead to large differences in the convolutional feature space. We introduce a lightweight de-stylization module that predicts parameters used for scaling and shifting feature maps to "undo" the changes incurred by filters.
  arXiv  Detail & Related papers  (2019-12-30T16:48:16Z)
• Very Long Natural Scenery Image Prediction by Outpainting [96.8509015981031]
  Outpainting receives less attention due to two challenges in it. First challenge is how to keep the spatial and content consistency between generated images and original input. Second challenge is how to maintain high quality in generated results.
  arXiv  Detail & Related papers  (2019-12-29T16:29:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.