MAT: Mask-Aware Transformer for Large Hole Image Inpainting

• URL: http://arxiv.org/abs/2203.15270v2
• Date: Wed, 30 Mar 2022 04:38:16 GMT
• Title: MAT: Mask-Aware Transformer for Large Hole Image Inpainting
• Authors: Wenbo Li, Zhe Lin, Kun Zhou, Lu Qi, Yi Wang, Jiaya Jia
• Abstract summary: We present a novel model for large hole inpainting, which unifies the merits of transformers and convolutions. Experiments demonstrate the state-of-the-art performance of the new model on multiple benchmark datasets.
• Score: 79.67039090195527
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent studies have shown the importance of modeling long-range interactions in the inpainting problem. To achieve this goal, existing approaches exploit either standalone attention techniques or transformers, but usually under a low resolution in consideration of computational cost. In this paper, we present a novel transformer-based model for large hole inpainting, which unifies the merits of transformers and convolutions to efficiently process high-resolution images. We carefully design each component of our framework to guarantee the high fidelity and diversity of recovered images. Specifically, we customize an inpainting-oriented transformer block, where the attention module aggregates non-local information only from partial valid tokens, indicated by a dynamic mask. Extensive experiments demonstrate the state-of-the-art performance of the new model on multiple benchmark datasets. Code is released at https://github.com/fenglinglwb/MAT.

Related papers

• HINT: High-quality INPainting Transformer with Mask-Aware Encoding and Enhanced Attention [14.055584700641212]
  Existing image inpainting methods leverage convolution-based downsampling approaches to reduce spatial dimensions. We propose an end-to-end High-quality INpainting Transformer, abbreviated as HINT, which consists of a novel mask-aware pixel-shuffle downsampling module. We demonstrate the superior performance of HINT compared to contemporary state-of-the-art models on four datasets.
  arXiv  Detail & Related papers  (2024-02-22T00:14:26Z)
• WavePaint: Resource-efficient Token-mixer for Self-supervised Inpainting [2.3014300466616078]
  This paper diverges from vision transformers by using a computationally-efficient WaveMix-based fully convolutional architecture -- WavePaint. It uses a 2D-discrete wavelet transform (DWT) for spatial and multi-resolution token-mixing along with convolutional layers. Our model even outperforms current GAN-based architectures in CelebA-HQ dataset without using an adversarially trainable discriminator.
  arXiv  Detail & Related papers  (2023-07-01T18:41:34Z)
• T-former: An Efficient Transformer for Image Inpainting [50.43302925662507]
  A class of attention-based network architectures, called transformer, has shown significant performance on natural language processing fields. In this paper, we design a novel attention linearly related to the resolution according to Taylor expansion, and based on this attention, a network called $T$-former is designed for image inpainting. Experiments on several benchmark datasets demonstrate that our proposed method achieves state-of-the-art accuracy while maintaining a relatively low number of parameters and computational complexity.
  arXiv  Detail & Related papers  (2023-05-12T04:10:42Z)
• DBAT: Dynamic Backward Attention Transformer for Material Segmentation with Cross-Resolution Patches [8.812837829361923]
  We propose the Dynamic Backward Attention Transformer (DBAT) to aggregate cross-resolution features. Experiments show that our DBAT achieves an accuracy of 86.85%, which is the best performance among state-of-the-art real-time models. We further align features to semantic labels, performing network dissection, to infer that the proposed model can extract material-related features better than other methods.
  arXiv  Detail & Related papers  (2023-05-06T03:47:20Z)
• Accurate Image Restoration with Attention Retractable Transformer [50.05204240159985]
  We propose Attention Retractable Transformer (ART) for image restoration. ART presents both dense and sparse attention modules in the network. We conduct extensive experiments on image super-resolution, denoising, and JPEG compression artifact reduction tasks.
  arXiv  Detail & Related papers  (2022-10-04T07:35:01Z)
• XCiT: Cross-Covariance Image Transformers [73.33400159139708]
  We propose a "transposed" version of self-attention that operates across feature channels rather than tokens. The resulting cross-covariance attention (XCA) has linear complexity in the number of tokens, and allows efficient processing of high-resolution images.
  arXiv  Detail & Related papers  (2021-06-17T17:33:35Z)
• Less is More: Pay Less Attention in Vision Transformers [61.05787583247392]
  Less attention vIsion Transformer builds upon the fact that convolutions, fully-connected layers, and self-attentions have almost equivalent mathematical expressions for processing image patch sequences. The proposed LIT achieves promising performance on image recognition tasks, including image classification, object detection and instance segmentation.
  arXiv  Detail & Related papers  (2021-05-29T05:26:07Z)
• Visual Saliency Transformer [127.33678448761599]
  We develop a novel unified model based on a pure transformer, Visual Saliency Transformer (VST), for both RGB and RGB-D salient object detection (SOD) It takes image patches as inputs and leverages the transformer to propagate global contexts among image patches. Experimental results show that our model outperforms existing state-of-the-art results on both RGB and RGB-D SOD benchmark datasets.
  arXiv  Detail & Related papers  (2021-04-25T08:24:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.