Efficient Neural Generation of 4K Masks for Homogeneous Diffusion Inpainting

• URL: http://arxiv.org/abs/2303.10096v2
• Date: Tue, 16 May 2023 08:33:04 GMT
• Title: Efficient Neural Generation of 4K Masks for Homogeneous Diffusion Inpainting
• Authors: Karl Schrader, Pascal Peter, Niklas K\"amper, Joachim Weickert
• Abstract summary: homogeneous diffusion inpainting can reconstruct images from sparse data with high quality. Mask optimisation for applications like image compression remains challenging. First neural approach for this so-called mask problem offered high speed and good quality for small images. We solve these problems and enable mask optimisation for high-resolution images through a neuroexplicit coarse-to-fine strategy.
• Score: 9.309874236223983
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With well-selected data, homogeneous diffusion inpainting can reconstruct images from sparse data with high quality. While 4K colour images of size 3840 x 2160 can already be inpainted in real time, optimising the known data for applications like image compression remains challenging: Widely used stochastic strategies can take days for a single 4K image. Recently, a first neural approach for this so-called mask optimisation problem offered high speed and good quality for small images. It trains a mask generation network with the help of a neural inpainting surrogate. However, these mask networks can only output masks for the resolution and mask density they were trained for. We solve these problems and enable mask optimisation for high-resolution images through a neuroexplicit coarse-to-fine strategy. Additionally, we improve the training and interpretability of mask networks by including a numerical inpainting solver directly into the network. This allows to generate masks for 4K images in around 0.6 seconds while exceeding the quality of stochastic methods on practically relevant densities. Compared to popular existing approaches, this is an acceleration of up to four orders of magnitude.

Related papers

• You Can Mask More For Extremely Low-Bitrate Image Compression [80.7692466922499]
  Learned image compression (LIC) methods have experienced significant progress during recent years. LIC methods fail to explicitly explore the image structure and texture components crucial for image compression. We present DA-Mask that samples visible patches based on the structure and texture of original images. We propose a simple yet effective masked compression model (MCM), the first framework that unifies LIC and LIC end-to-end for extremely low-bitrate compression.
  arXiv  Detail & Related papers  (2023-06-27T15:36:22Z)
• CoordFill: Efficient High-Resolution Image Inpainting via Parameterized Coordinate Querying [52.91778151771145]
  In this paper, we try to break the limitations for the first time thanks to the recent development of continuous implicit representation. Experiments show that the proposed method achieves real-time performance on the 2048$times$2048 images using a single GTX 2080 Ti GPU.
  arXiv  Detail & Related papers  (2023-03-15T11:13:51Z)
• MP-Former: Mask-Piloted Transformer for Image Segmentation [16.620469868310288]
  Mask2Former suffers from inconsistent mask predictions between decoder layers. We propose a mask-piloted training approach, which feeds noised ground-truth masks in masked-attention and trains the model to reconstruct the original ones.
  arXiv  Detail & Related papers  (2023-03-13T17:57:59Z)
• Improving Masked Autoencoders by Learning Where to Mask [65.89510231743692]
  Masked image modeling is a promising self-supervised learning method for visual data. We present AutoMAE, a framework that uses Gumbel-Softmax to interlink an adversarially-trained mask generator and a mask-guided image modeling process. In our experiments, AutoMAE is shown to provide effective pretraining models on standard self-supervised benchmarks and downstream tasks.
  arXiv  Detail & Related papers  (2023-03-12T05:28:55Z)
• RePaint: Inpainting using Denoising Diffusion Probabilistic Models [161.74792336127345]
  Free-form inpainting is the task of adding new content to an image in the regions specified by an arbitrary binary mask. We propose RePaint: A Denoising Probabilistic Model (DDPM) based inpainting approach that is applicable to even extreme masks. We validate our method for both faces and general-purpose image inpainting using standard and extreme masks.
  arXiv  Detail & Related papers  (2022-01-24T18:40:15Z)
• Learning Sparse Masks for Diffusion-based Image Inpainting [10.633099921979674]
  Diffusion-based inpainting is a powerful tool for the reconstruction of images from sparse data. We provide a model for highly efficient adaptive mask generation. Experiments indicate that our model can achieve competitive quality with an acceleration by as much as four orders of magnitude.
  arXiv  Detail & Related papers  (2021-10-06T10:20:59Z)
• Iterative Facial Image Inpainting using Cyclic Reverse Generator [0.913755431537592]
  Cyclic Reverse Generator (CRG) architecture provides an encoder-generator model. We empirically observed that only a few iterations are sufficient to generate realistic images with the proposed model. Our method allows applying sketch-based inpaintings, using variety of mask types, and producing multiple and diverse results.
  arXiv  Detail & Related papers  (2021-01-18T12:19:58Z)
• DCT-Mask: Discrete Cosine Transform Mask Representation for Instance Segmentation [50.70679435176346]
  We propose a new mask representation by applying the discrete cosine transform(DCT) to encode the high-resolution binary grid mask into a compact vector. Our method, termed DCT-Mask, could be easily integrated into most pixel-based instance segmentation methods.
  arXiv  Detail & Related papers  (2020-11-19T15:00:21Z)
• Free-Form Image Inpainting via Contrastive Attention Network [64.05544199212831]
  In image inpainting tasks, masks with any shapes can appear anywhere in images which form complex patterns. It is difficult for encoders to capture such powerful representations under this complex situation. We propose a self-supervised Siamese inference network to improve the robustness and generalization.
  arXiv  Detail & Related papers  (2020-10-29T14:46:05Z)
• Fully Convolutional Networks for Automatically Generating Image Masks to Train Mask R-CNN [4.901462756978097]
  The Mask R-CNN method achieves the best results in object detection until now, however, it is very time-consuming and laborious to get the object Masks for training. This paper proposes a novel automatically generating image masks method for the state-of-the-art Mask R-CNN deep learning method. Our proposed method can obtain the image masks automatically to train Mask R-CNN, and it can achieve very high classification accuracy with an over 90% mean of average precision (mAP) for segmentation.
  arXiv  Detail & Related papers  (2020-03-03T08:09:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.