Cross-View Image Synthesis with Deformable Convolution and Attention Mechanism

• URL: http://arxiv.org/abs/2007.09858v1
• Date: Mon, 20 Jul 2020 03:08:36 GMT
• Title: Cross-View Image Synthesis with Deformable Convolution and Attention Mechanism
• Authors: Hao Ding, Songsong Wu, Hao Tang, Fei Wu, Guangwei Gao and Xiao-Yuan Jing
• Abstract summary: We propose to use Generative Adversarial Networks(GANs) based on a deformable convolution and attention mechanism to solve the problem of cross-view image synthesis. It is difficult to understand and transform scenes appearance and semantic information from another view, thus we use deformed convolution in the U-net network to improve the network's ability to extract features of objects at different scales.
• Score: 29.528402825356398
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Learning to generate natural scenes has always been a daunting task in computer vision. This is even more laborious when generating images with very different views. When the views are very different, the view fields have little overlap or objects are occluded, leading the task very challenging. In this paper, we propose to use Generative Adversarial Networks(GANs) based on a deformable convolution and attention mechanism to solve the problem of cross-view image synthesis (see Fig.1). It is difficult to understand and transform scenes appearance and semantic information from another view, thus we use deformed convolution in the U-net network to improve the network's ability to extract features of objects at different scales. Moreover, to better learn the correspondence between images from different views, we apply an attention mechanism to refine the intermediate feature map thus generating more realistic images. A large number of experiments on different size images on the Dayton dataset[1] show that our model can produce better results than state-of-the-art methods.

Related papers

• Cross-Image Attention for Zero-Shot Appearance Transfer [68.43651329067393]
  We introduce a cross-image attention mechanism that implicitly establishes semantic correspondences across images. We harness three mechanisms that either manipulate the noisy latent codes or the model's internal representations throughout the denoising process. Experiments show that our method is effective across a wide range of object categories and is robust to variations in shape, size, and viewpoint.
  arXiv  Detail & Related papers  (2023-11-06T18:33:24Z)
• GM-NeRF: Learning Generalizable Model-based Neural Radiance Fields from Multi-view Images [79.39247661907397]
  We introduce an effective framework Generalizable Model-based Neural Radiance Fields to synthesize free-viewpoint images. Specifically, we propose a geometry-guided attention mechanism to register the appearance code from multi-view 2D images to a geometry proxy.
  arXiv  Detail & Related papers  (2023-03-24T03:32:02Z)
• Tuning computer vision models with task rewards [88.45787930908102]
  Misalignment between model predictions and intended usage can be detrimental for the deployment of computer vision models. In natural language processing, this is often addressed using reinforcement learning techniques that align models with a task reward. We adopt this approach and show its surprising effectiveness across multiple computer vision tasks, such as object detection, panoptic segmentation, colorization and image captioning.
  arXiv  Detail & Related papers  (2023-02-16T11:49:48Z)
• Dual Pyramid Generative Adversarial Networks for Semantic Image Synthesis [94.76988562653845]
  The goal of semantic image synthesis is to generate photo-realistic images from semantic label maps. Current state-of-the-art approaches, however, still struggle to generate realistic objects in images at various scales. We propose a Dual Pyramid Generative Adversarial Network (DP-GAN) that learns the conditioning of spatially-adaptive normalization blocks at all scales jointly.
  arXiv  Detail & Related papers  (2022-10-08T18:45:44Z)
• Generative Modeling for Multi-task Visual Learning [40.96212750592383]
  We consider a novel problem of learning a shared generative model that is useful across various visual perception tasks. We propose a general multi-task oriented generative modeling framework, by coupling a discriminative multi-task network with a generative network. Our framework consistently outperforms state-of-the-art multi-task approaches.
  arXiv  Detail & Related papers  (2021-06-25T03:42:59Z)
• Self-Supervised Representation Learning from Flow Equivariance [97.13056332559526]
  We present a new self-supervised learning representation framework that can be directly deployed on a video stream of complex scenes. Our representations, learned from high-resolution raw video, can be readily used for downstream tasks on static images.
  arXiv  Detail & Related papers  (2021-01-16T23:44:09Z)
• Fine-grained Image-to-Image Transformation towards Visual Recognition [102.51124181873101]
  We aim at transforming an image with a fine-grained category to synthesize new images that preserve the identity of the input image. We adopt a model based on generative adversarial networks to disentangle the identity related and unrelated factors of an image. Experiments on the CompCars and Multi-PIE datasets demonstrate that our model preserves the identity of the generated images much better than the state-of-the-art image-to-image transformation models.
  arXiv  Detail & Related papers  (2020-01-12T05:26:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.