Feature Refinement to Improve High Resolution Image Inpainting
- URL: http://arxiv.org/abs/2206.13644v2
- Date: Wed, 29 Jun 2022 18:16:30 GMT
- Title: Feature Refinement to Improve High Resolution Image Inpainting
- Authors: Prakhar Kulshreshtha, Brian Pugh and Salma Jiddi
- Abstract summary: Inpainting networks are often unable to generate globally coherent structures at resolutions higher than their training set.
We optimize the intermediate featuremaps of a network by minimizing a multiscale consistency loss at inference.
This runtime optimization improves the inpainting results and establishes a new state-of-the-art for high resolution inpainting.
- Score: 1.4824891788575418
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: In this paper, we address the problem of degradation in inpainting quality of
neural networks operating at high resolutions. Inpainting networks are often
unable to generate globally coherent structures at resolutions higher than
their training set. This is partially attributed to the receptive field
remaining static, despite an increase in image resolution. Although downscaling
the image prior to inpainting produces coherent structure, it inherently lacks
detail present at higher resolutions. To get the best of both worlds, we
optimize the intermediate featuremaps of a network by minimizing a multiscale
consistency loss at inference. This runtime optimization improves the
inpainting results and establishes a new state-of-the-art for high resolution
inpainting. Code is available at:
https://github.com/geomagical/lama-with-refiner/tree/refinement.
Related papers
- T-former: An Efficient Transformer for Image Inpainting [50.43302925662507]
A class of attention-based network architectures, called transformer, has shown significant performance on natural language processing fields.
In this paper, we design a novel attention linearly related to the resolution according to Taylor expansion, and based on this attention, a network called $T$-former is designed for image inpainting.
Experiments on several benchmark datasets demonstrate that our proposed method achieves state-of-the-art accuracy while maintaining a relatively low number of parameters and computational complexity.
arXiv Detail & Related papers (2023-05-12T04:10:42Z) - CoordFill: Efficient High-Resolution Image Inpainting via Parameterized
Coordinate Querying [52.91778151771145]
In this paper, we try to break the limitations for the first time thanks to the recent development of continuous implicit representation.
Experiments show that the proposed method achieves real-time performance on the 2048$times$2048 images using a single GTX 2080 Ti GPU.
arXiv Detail & Related papers (2023-03-15T11:13:51Z) - A Wasserstein GAN for Joint Learning of Inpainting and its Spatial
Optimisation [3.4392739159262145]
We propose the first generative adversarial network for spatial inpainting data optimisation.
In contrast to previous approaches, it allows joint training of an inpainting generator and a corresponding mask network.
This yields significant improvements in visual quality and speed over conventional models and also outperforms current optimisation networks.
arXiv Detail & Related papers (2022-02-11T14:02:36Z) - Resolution-robust Large Mask Inpainting with Fourier Convolutions [10.152370311844445]
Inpainting systems often struggle with large missing areas, complex geometric structures, and high-resolution images.
We find that one of the main reasons for that is the lack of an effective receptive field in both the inpainting network and the loss function.
We propose a new method called large mask inpainting (LaMa) to alleviate this issue.
arXiv Detail & Related papers (2021-09-15T08:54:29Z) - Spatially-Adaptive Image Restoration using Distortion-Guided Networks [51.89245800461537]
We present a learning-based solution for restoring images suffering from spatially-varying degradations.
We propose SPAIR, a network design that harnesses distortion-localization information and dynamically adjusts to difficult regions in the image.
arXiv Detail & Related papers (2021-08-19T11:02:25Z) - InfinityGAN: Towards Infinite-Resolution Image Synthesis [92.40782797030977]
We present InfinityGAN, a method to generate arbitrary-resolution images.
We show how it trains and infers patch-by-patch seamlessly with low computational resources.
arXiv Detail & Related papers (2021-04-08T17:59:30Z) - Zoom-to-Inpaint: Image Inpainting with High-Frequency Details [39.582275854002994]
We propose applying super-resolution to coarsely reconstructed outputs, refining them at high resolution, and then downscaling the output to the original resolution.
By introducing high-resolution images to the refinement network, our framework is able to reconstruct finer details that are usually smoothed out due to spectral bias.
Our zoom-in, refine and zoom-out strategy, combined with high-resolution supervision and progressive learning, constitutes a framework-agnostic approach for enhancing high-frequency details.
arXiv Detail & Related papers (2020-12-17T05:39:37Z) - Contextual Residual Aggregation for Ultra High-Resolution Image
Inpainting [12.839962012888199]
We propose a Contextual Residual Aggregation (CRA) mechanism that can produce high-frequency residuals for missing contents.
CRA mechanism produces high-frequency residuals for missing contents by weighted aggregating residuals from contextual patches.
We train the proposed model on small images with resolutions 512x512 and perform inference on high-resolution images, achieving compelling inpainting quality.
arXiv Detail & Related papers (2020-05-19T18:55:32Z) - Gated Fusion Network for Degraded Image Super Resolution [78.67168802945069]
We propose a dual-branch convolutional neural network to extract base features and recovered features separately.
By decomposing the feature extraction step into two task-independent streams, the dual-branch model can facilitate the training process.
arXiv Detail & Related papers (2020-03-02T13:28:32Z) - Image Fine-grained Inpainting [89.17316318927621]
We present a one-stage model that utilizes dense combinations of dilated convolutions to obtain larger and more effective receptive fields.
To better train this efficient generator, except for frequently-used VGG feature matching loss, we design a novel self-guided regression loss.
We also employ a discriminator with local and global branches to ensure local-global contents consistency.
arXiv Detail & Related papers (2020-02-07T03:45:25Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.