Related papers: Generalised Image Outpainting with U-Transformer

Generalised Image Outpainting with U-Transformer

URL: http://arxiv.org/abs/2201.11403v1
Date: Thu, 27 Jan 2022 09:41:58 GMT
Title: Generalised Image Outpainting with U-Transformer
Authors: Penglei Gao, Xi Yang, Rui Zhang, Kaizhu Huang, and Yujie Geng
Abstract summary: We develop a novel transformer-based generative adversarial network called U-Transformer. Specifically, we design a generator as an encoder-to-decoder structure embedded with the popular Swin Transformer blocks. We experimentally demonstrate that our proposed method could produce visually appealing results for generalised image outpainting.
Score: 19.894445491176878
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
Abstract: While most present image outpainting conducts horizontal extrapolation, we study the generalised image outpainting problem that extrapolates visual context all-side around a given image. To this end, we develop a novel transformer-based generative adversarial network called U-Transformer able to extend image borders with plausible structure and details even for complicated scenery images. Specifically, we design a generator as an encoder-to-decoder structure embedded with the popular Swin Transformer blocks. As such, our novel framework can better cope with image long-range dependencies which are crucially important for generalised image outpainting. We propose additionally a U-shaped structure and multi-view Temporal Spatial Predictor network to reinforce image self-reconstruction as well as unknown-part prediction smoothly and realistically. We experimentally demonstrate that our proposed method could produce visually appealing results for generalized image outpainting against the state-of-the-art image outpainting approaches.

Related papers

Outpainting by Queries [23.626012684754965]
We propose a novel hybrid vision-transformer-based encoder-decoder framework, named textbfQuery textbfOutpainting textbfTRansformer (textbfQueryOTR) We experimentally show that QueryOTR could generate visually appealing results smoothly and realistically against the state-of-the-art image outpainting approaches.
arXiv Detail & Related papers (2022-07-12T04:48:41Z)
Cylin-Painting: Seamless {360\textdegree} Panoramic Image Outpainting and Beyond [136.18504104345453]
We present a Cylin-Painting framework that involves meaningful collaborations between inpainting and outpainting. The proposed algorithm can be effectively extended to other panoramic vision tasks, such as object detection, depth estimation, and image super-resolution.
arXiv Detail & Related papers (2022-04-18T21:18:49Z)
MAT: Mask-Aware Transformer for Large Hole Image Inpainting [79.67039090195527]
We present a novel model for large hole inpainting, which unifies the merits of transformers and convolutions. Experiments demonstrate the state-of-the-art performance of the new model on multiple benchmark datasets.
arXiv Detail & Related papers (2022-03-29T06:36:17Z)
PPT Fusion: Pyramid Patch Transformerfor a Case Study in Image Fusion [37.993611194758195]
We propose a Patch PyramidTransformer(PPT) to address the issues of extracting semantic information from an image. The experimental results demonstrate its superior performance against the state-of-the-art fusion approaches.
arXiv Detail & Related papers (2021-07-29T13:57:45Z)
Structure First Detail Next: Image Inpainting with Pyramid Generator [26.94101909283021]
We propose to build a Pyramid Generator by stacking several sub-generators. Lower-layer sub-generators focus on restoring image structures while the higher-layer sub-generators emphasize image details. Our approach has a learning scheme of progressively increasing hole size, which allows it to restore large-hole images.
arXiv Detail & Related papers (2021-06-16T16:00:16Z)
Diverse Image Inpainting with Bidirectional and Autoregressive Transformers [55.21000775547243]
We propose BAT-Fill, an image inpainting framework with a novel bidirectional autoregressive transformer (BAT) BAT-Fill inherits the merits of transformers and CNNs in a two-stage manner, which allows to generate high-resolution contents without being constrained by the quadratic complexity of attention in transformers.
arXiv Detail & Related papers (2021-04-26T03:52:27Z)
In&Out : Diverse Image Outpainting via GAN Inversion [89.84841983778672]
Image outpainting seeks for a semantically consistent extension of the input image beyond its available content. In this work, we formulate the problem from the perspective of inverting generative adversarial networks. Our generator renders micro-patches conditioned on their joint latent code as well as their individual positions in the image.
arXiv Detail & Related papers (2021-04-01T17:59:10Z)
Free-Form Image Inpainting via Contrastive Attention Network [64.05544199212831]
In image inpainting tasks, masks with any shapes can appear anywhere in images which form complex patterns. It is difficult for encoders to capture such powerful representations under this complex situation. We propose a self-supervised Siamese inference network to improve the robustness and generalization.
arXiv Detail & Related papers (2020-10-29T14:46:05Z)
Generating Person Images with Appearance-aware Pose Stylizer [66.44220388377596]
We present a novel end-to-end framework to generate realistic person images based on given person poses and appearances. The core of our framework is a novel generator called Appearance-aware Pose Stylizer (APS) which generates human images by coupling the target pose with the conditioned person appearance progressively.
arXiv Detail & Related papers (2020-07-17T15:58:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.