Image Inpainting with Edge-guided Learnable Bidirectional Attention Maps

• URL: http://arxiv.org/abs/2104.12087v1
• Date: Sun, 25 Apr 2021 07:25:16 GMT
• Title: Image Inpainting with Edge-guided Learnable Bidirectional Attention Maps
• Authors: Dongsheng Wang, Chaohao Xie, Shaohui Liu, Zhenxing Niu, Wangmeng Zuo
• Abstract summary: We present an edge-guided learnable bidirectional attention map (Edge-LBAM) for improving image inpainting of irregular holes. Our Edge-LBAM method contains dual procedures,including structure-aware mask-updating guided by predict edges. Extensive experiments show that our Edge-LBAM is effective in generating coherent image structures and preventing color discrepancy and blurriness.
• Score: 85.67745220834718
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: For image inpainting, the convolutional neural networks (CNN) in previous methods often adopt standard convolutional operator, which treats valid pixels and holes indistinguishably. As a result, they are limited in handling irregular holes and tend to produce color-discrepant and blurry inpainting result. Partial convolution (PConv) copes with this issue by conducting masked convolution and feature re-normalization conditioned only on valid pixels, but the mask-updating is handcrafted and independent with image structural information. In this paper, we present an edge-guided learnable bidirectional attention map (Edge-LBAM) for improving image inpainting of irregular holes with several distinct merits. Instead of using a hard 0-1 mask, a learnable attention map module is introduced for learning feature re-normalization and mask-updating in an end-to-end manner. Learnable reverse attention maps are further proposed in the decoder for emphasizing on filling in unknown pixels instead of reconstructing all pixels. Motivated by that the filling-in order is crucial to inpainting results and largely depends on image structures in exemplar-based methods, we further suggest a multi-scale edge completion network to predict coherent edges. Our Edge-LBAM method contains dual procedures,including structure-aware mask-updating guided by predict edges and attention maps generated by masks for feature re-normalization.Extensive experiments show that our Edge-LBAM is effective in generating coherent image structures and preventing color discrepancy and blurriness, and performs favorably against the state-of-the-art methods in terms of qualitative metrics and visual quality.

Related papers

• Parallax-Tolerant Unsupervised Deep Image Stitching [57.76737888499145]
  We propose UDIS++, a parallax-tolerant unsupervised deep image stitching technique. First, we propose a robust and flexible warp to model the image registration from global homography to local thin-plate spline motion. To further eliminate the parallax artifacts, we propose to composite the stitched image seamlessly by unsupervised learning for seam-driven composition masks.
  arXiv  Detail & Related papers  (2023-02-16T10:40:55Z)
• MaskSketch: Unpaired Structure-guided Masked Image Generation [56.88038469743742]
  MaskSketch is an image generation method that allows spatial conditioning of the generation result using a guiding sketch as an extra conditioning signal during sampling. We show that intermediate self-attention maps of a masked generative transformer encode important structural information of the input image. Our results show that MaskSketch achieves high image realism and fidelity to the guiding structure.
  arXiv  Detail & Related papers  (2023-02-10T20:27:02Z)
• Reference Guided Image Inpainting using Facial Attributes [18.09748108912419]
  We propose an alternative user-guided inpainting architecture that manipulates facial attributes using a single reference image as the guide. Our end-to-end model consists of attribute extractors for accurate reference image attribute transfer and an inpainting model to map the attributes realistically and accurately.
  arXiv  Detail & Related papers  (2023-01-19T12:39:08Z)
• Unbiased Multi-Modality Guidance for Image Inpainting [27.286351511243502]
  We develop an end-to-end multi-modality guided transformer network for image inpainting. Within each transformer block, the proposed spatial-aware attention module can learn the multi-modal structural features efficiently. Our method enriches semantically consistent context in an image based on discriminative information from multiple modalities.
  arXiv  Detail & Related papers  (2022-08-25T03:13:43Z)
• What You See is What You Classify: Black Box Attributions [61.998683569022006]
  We train a deep network, the Explainer, to predict attributions for a pre-trained black-box classifier, the Explanandum. Unlike most existing approaches, ours is capable of directly generating very distinct class-specific masks. We show that our attributions are superior to established methods both visually and quantitatively.
  arXiv  Detail & Related papers  (2022-05-23T12:30:04Z)
• One-Stage Deep Edge Detection Based on Dense-Scale Feature Fusion and Pixel-Level Imbalance Learning [5.370848116287344]
  We propose a one-stage neural network model that can generate high-quality edge images without postprocessing. The proposed model adopts a classic encoder-decoder framework in which a pre-trained neural model is used as the encoder. We propose a new loss function that addresses the pixel-level imbalance in the edge image.
  arXiv  Detail & Related papers  (2022-03-17T15:26:00Z)
• Deep Rectangling for Image Stitching: A Learning Baseline [57.76737888499145]
  We build the first image stitching rectangling dataset with a large diversity in irregular boundaries and scenes. Experiments demonstrate our superiority over traditional methods both quantitatively and qualitatively.
  arXiv  Detail & Related papers  (2022-03-08T03:34:10Z)
• Image Generation with Self Pixel-wise Normalization [17.147675335268282]
  Region-adaptive normalization (RAN) methods have been widely used in the generative adversarial network (GAN)-based image-to-image translation technique. This paper presents a novel normalization method, called self pixel-wise normalization (SPN), which effectively boosts the generative performance by performing the pixel-adaptive affine transformation without the mask image.
  arXiv  Detail & Related papers  (2022-01-26T03:14:31Z)
• Image Inpainting by End-to-End Cascaded Refinement with Mask Awareness [66.55719330810547]
  Inpainting arbitrary missing regions is challenging because learning valid features for various masked regions is nontrivial. We propose a novel mask-aware inpainting solution that learns multi-scale features for missing regions in the encoding phase. Our framework is validated both quantitatively and qualitatively via extensive experiments on three public datasets.
  arXiv  Detail & Related papers  (2021-04-28T13:17:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.