Improve Deep Image Inpainting by Emphasizing the Complexity of Missing Regions

• URL: http://arxiv.org/abs/2202.06266v1
• Date: Sun, 13 Feb 2022 09:14:52 GMT
• Title: Improve Deep Image Inpainting by Emphasizing the Complexity of Missing Regions
• Authors: Yufeng Wang, Dan Li, Cong Xu and Min Yang
• Abstract summary: In this paper, we enhance the deep image inpainting models with the help of classical image complexity metrics. A knowledge-assisted index composed of missingness complexity and forward loss is presented to guide the batch selection in the training procedure. We experimentally demonstrate the improvements for several recently developed image inpainting models on various datasets.
• Score: 20.245637164975594
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep image inpainting research mainly focuses on constructing various neural network architectures or imposing novel optimization objectives. However, on the one hand, building a state-of-the-art deep inpainting model is an extremely complex task, and on the other hand, the resulting performance gains are sometimes very limited. We believe that besides the frameworks of inpainting models, lightweight traditional image processing techniques, which are often overlooked, can actually be helpful to these deep models. In this paper, we enhance the deep image inpainting models with the help of classical image complexity metrics. A knowledge-assisted index composed of missingness complexity and forward loss is presented to guide the batch selection in the training procedure. This index helps find samples that are more conducive to optimization in each iteration and ultimately boost the overall inpainting performance. The proposed approach is simple and can be plugged into many deep inpainting models by changing only a few lines of code. We experimentally demonstrate the improvements for several recently developed image inpainting models on various datasets.

Related papers

• PrefPaint: Aligning Image Inpainting Diffusion Model with Human Preference [62.72779589895124]
  We make the first attempt to align diffusion models for image inpainting with human aesthetic standards via a reinforcement learning framework. We train a reward model with a dataset we construct, consisting of nearly 51,000 images annotated with human preferences. Experiments on inpainting comparison and downstream tasks, such as image extension and 3D reconstruction, demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2024-10-29T11:49:39Z)
• Segmentation-Based Parametric Painting [22.967620358813214]
  We introduce a novel image-to-painting method that facilitates the creation of large-scale, high-fidelity paintings with human-like quality and stylistic variation. We introduce a segmentation-based painting process and a dynamic attention map approach inspired by human painting strategies. Our optimized batch processing and patch-based loss framework enable efficient handling of large canvases.
  arXiv  Detail & Related papers  (2023-11-24T04:15:10Z)
• T-former: An Efficient Transformer for Image Inpainting [50.43302925662507]
  A class of attention-based network architectures, called transformer, has shown significant performance on natural language processing fields. In this paper, we design a novel attention linearly related to the resolution according to Taylor expansion, and based on this attention, a network called $T$-former is designed for image inpainting. Experiments on several benchmark datasets demonstrate that our proposed method achieves state-of-the-art accuracy while maintaining a relatively low number of parameters and computational complexity.
  arXiv  Detail & Related papers  (2023-05-12T04:10:42Z)
• GRIG: Few-Shot Generative Residual Image Inpainting [27.252855062283825]
  We present a novel few-shot generative residual image inpainting method that produces high-quality inpainting results. The core idea is to propose an iterative residual reasoning method that incorporates Convolutional Neural Networks (CNNs) for feature extraction. We also propose a novel forgery-patch adversarial training strategy to create faithful textures and detailed appearances.
  arXiv  Detail & Related papers  (2023-04-24T12:19:06Z)
• Perceptual Artifacts Localization for Inpainting [60.5659086595901]
  We propose a new learning task of automatic segmentation of inpainting perceptual artifacts. We train advanced segmentation networks on a dataset to reliably localize inpainting artifacts within inpainted images. We also propose a new evaluation metric called Perceptual Artifact Ratio (PAR), which is the ratio of objectionable inpainted regions to the entire inpainted area.
  arXiv  Detail & Related papers  (2022-08-05T18:50:51Z)
• MAT: Mask-Aware Transformer for Large Hole Image Inpainting [79.67039090195527]
  We present a novel model for large hole inpainting, which unifies the merits of transformers and convolutions. Experiments demonstrate the state-of-the-art performance of the new model on multiple benchmark datasets.
  arXiv  Detail & Related papers  (2022-03-29T06:36:17Z)
• Incremental Transformer Structure Enhanced Image Inpainting with Masking Positional Encoding [38.014569953980754]
  The proposed model restores holistic image structures with a powerful attention-based transformer model in a fixed low-resolution sketch space. Our model can be integrated with other pretrained inpainting models efficiently with the zero-d residual addition.
  arXiv  Detail & Related papers  (2022-03-02T04:27:27Z)
• A Wasserstein GAN for Joint Learning of Inpainting and its Spatial Optimisation [3.4392739159262145]
  We propose the first generative adversarial network for spatial inpainting data optimisation. In contrast to previous approaches, it allows joint training of an inpainting generator and a corresponding mask network. This yields significant improvements in visual quality and speed over conventional models and also outperforms current optimisation networks.
  arXiv  Detail & Related papers  (2022-02-11T14:02:36Z)
• RigNet: Repetitive Image Guided Network for Depth Completion [20.66405067066299]
  Recent approaches mainly focus on image guided learning to predict dense results. blurry image guidance and object structures in depth still impede the performance of image guided frameworks. We explore a repetitive design in our image guided network to sufficiently and gradually recover depth values. Our method achieves state-of-the-art result on the NYUv2 dataset and ranks 1st on the KITTI benchmark at the time of submission.
  arXiv  Detail & Related papers  (2021-07-29T08:00:33Z)
• Deep Image Compositing [93.75358242750752]
  We propose a new method which can automatically generate high-quality image composites without any user input. Inspired by Laplacian pyramid blending, a dense-connected multi-stream fusion network is proposed to effectively fuse the information from the foreground and background images. Experiments show that the proposed method can automatically generate high-quality composites and outperforms existing methods both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2020-11-04T06:12:24Z)
• Learning End-to-End Lossy Image Compression: A Benchmark [90.35363142246806]
  We first conduct a comprehensive literature survey of learned image compression methods. We describe milestones in cutting-edge learned image-compression methods, review a broad range of existing works, and provide insights into their historical development routes. By introducing a coarse-to-fine hyperprior model for entropy estimation and signal reconstruction, we achieve improved rate-distortion performance.
  arXiv  Detail & Related papers  (2020-02-10T13:13:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.