Learning Sparse Masks for Diffusion-based Image Inpainting
- URL: http://arxiv.org/abs/2110.02636v1
- Date: Wed, 6 Oct 2021 10:20:59 GMT
- Title: Learning Sparse Masks for Diffusion-based Image Inpainting
- Authors: Tobias Alt, Pascal Peter, Joachim Weickert
- Abstract summary: Diffusion-based inpainting is a powerful tool for the reconstruction of images from sparse data.
We provide a model for highly efficient adaptive mask generation.
Experiments indicate that our model can achieve competitive quality with an acceleration by as much as four orders of magnitude.
- Score: 10.633099921979674
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Diffusion-based inpainting is a powerful tool for the reconstruction of
images from sparse data. Its quality strongly depends on the choice of known
data. Optimising their spatial location -- the inpainting mask -- is
challenging. A commonly used tool for this task are stochastic optimisation
strategies. However, they are slow as they compute multiple inpainting results.
We provide a remedy in terms of a learned mask generation model. By emulating
the complete inpainting pipeline with two networks for mask generation and
neural surrogate inpainting, we obtain a model for highly efficient adaptive
mask generation. Experiments indicate that our model can achieve competitive
quality with an acceleration by as much as four orders of magnitude. Our
findings serve as a basis for making diffusion-based inpainting more attractive
for various applications such as image compression, where fast encoding is
highly desirable.
Related papers
- ColorMAE: Exploring data-independent masking strategies in Masked AutoEncoders [53.3185750528969]
Masked AutoEncoders (MAE) have emerged as a robust self-supervised framework.
We introduce a data-independent method, termed ColorMAE, which generates different binary mask patterns by filtering random noise.
We demonstrate our strategy's superiority in downstream tasks compared to random masking.
arXiv Detail & Related papers (2024-07-17T22:04:00Z) - DiffGANPaint: Fast Inpainting Using Denoising Diffusion GANs [19.690288425689328]
In this paper, we propose a Denoising Diffusion Probabilistic Model (DDPM) based model capable of filling missing pixels fast.
Experiments on general-purpose image inpainting datasets verify that our approach performs superior or on par with most contemporary works.
arXiv Detail & Related papers (2023-08-03T17:50:41Z) - Efficient Neural Generation of 4K Masks for Homogeneous Diffusion
Inpainting [9.309874236223983]
homogeneous diffusion inpainting can reconstruct images from sparse data with high quality.
Mask optimisation for applications like image compression remains challenging.
First neural approach for this so-called mask problem offered high speed and good quality for small images.
We solve these problems and enable mask optimisation for high-resolution images through a neuroexplicit coarse-to-fine strategy.
arXiv Detail & Related papers (2023-03-17T16:16:36Z) - Improving Masked Autoencoders by Learning Where to Mask [65.89510231743692]
Masked image modeling is a promising self-supervised learning method for visual data.
We present AutoMAE, a framework that uses Gumbel-Softmax to interlink an adversarially-trained mask generator and a mask-guided image modeling process.
In our experiments, AutoMAE is shown to provide effective pretraining models on standard self-supervised benchmarks and downstream tasks.
arXiv Detail & Related papers (2023-03-12T05:28:55Z) - Shape-Aware Masking for Inpainting in Medical Imaging [49.61617087640379]
Inpainting has been proposed as a successful deep learning technique for unsupervised medical image model discovery.
We introduce a method for generating shape-aware masks for inpainting, which aims at learning the statistical shape prior.
We propose an unsupervised guided masking approach based on an off-the-shelf inpainting model and a superpixel over-segmentation algorithm.
arXiv Detail & Related papers (2022-07-12T18:35:17Z) - MAT: Mask-Aware Transformer for Large Hole Image Inpainting [79.67039090195527]
We present a novel model for large hole inpainting, which unifies the merits of transformers and convolutions.
Experiments demonstrate the state-of-the-art performance of the new model on multiple benchmark datasets.
arXiv Detail & Related papers (2022-03-29T06:36:17Z) - A Wasserstein GAN for Joint Learning of Inpainting and its Spatial
Optimisation [3.4392739159262145]
We propose the first generative adversarial network for spatial inpainting data optimisation.
In contrast to previous approaches, it allows joint training of an inpainting generator and a corresponding mask network.
This yields significant improvements in visual quality and speed over conventional models and also outperforms current optimisation networks.
arXiv Detail & Related papers (2022-02-11T14:02:36Z) - RePaint: Inpainting using Denoising Diffusion Probabilistic Models [161.74792336127345]
Free-form inpainting is the task of adding new content to an image in the regions specified by an arbitrary binary mask.
We propose RePaint: A Denoising Probabilistic Model (DDPM) based inpainting approach that is applicable to even extreme masks.
We validate our method for both faces and general-purpose image inpainting using standard and extreme masks.
arXiv Detail & Related papers (2022-01-24T18:40:15Z) - Iterative Facial Image Inpainting using Cyclic Reverse Generator [0.913755431537592]
Cyclic Reverse Generator (CRG) architecture provides an encoder-generator model.
We empirically observed that only a few iterations are sufficient to generate realistic images with the proposed model.
Our method allows applying sketch-based inpaintings, using variety of mask types, and producing multiple and diverse results.
arXiv Detail & Related papers (2021-01-18T12:19:58Z) - Free-Form Image Inpainting via Contrastive Attention Network [64.05544199212831]
In image inpainting tasks, masks with any shapes can appear anywhere in images which form complex patterns.
It is difficult for encoders to capture such powerful representations under this complex situation.
We propose a self-supervised Siamese inference network to improve the robustness and generalization.
arXiv Detail & Related papers (2020-10-29T14:46:05Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.