Wavelet-based Unsupervised Label-to-Image Translation

• URL: http://arxiv.org/abs/2305.09647v1
• Date: Tue, 16 May 2023 17:48:44 GMT
• Title: Wavelet-based Unsupervised Label-to-Image Translation
• Authors: George Eskandar, Mohamed Abdelsamad, Karim Armanious, Shuai Zhang, Bin Yang
• Abstract summary: We propose a new Unsupervised paradigm for SIS (USIS) that makes use of a self-supervised segmentation loss and whole image wavelet based discrimination. We test our methodology on 3 challenging datasets and demonstrate its ability to bridge the performance gap between paired and unpaired models.
• Score: 9.339522647331334
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Semantic Image Synthesis (SIS) is a subclass of image-to-image translation where a semantic layout is used to generate a photorealistic image. State-of-the-art conditional Generative Adversarial Networks (GANs) need a huge amount of paired data to accomplish this task while generic unpaired image-to-image translation frameworks underperform in comparison, because they color-code semantic layouts and learn correspondences in appearance instead of semantic content. Starting from the assumption that a high quality generated image should be segmented back to its semantic layout, we propose a new Unsupervised paradigm for SIS (USIS) that makes use of a self-supervised segmentation loss and whole image wavelet based discrimination. Furthermore, in order to match the high-frequency distribution of real images, a novel generator architecture in the wavelet domain is proposed. We test our methodology on 3 challenging datasets and demonstrate its ability to bridge the performance gap between paired and unpaired models.

Related papers

• Unlocking Pre-trained Image Backbones for Semantic Image Synthesis [29.688029979801577]
  We propose a new class of GAN discriminators for semantic image synthesis that generates highly realistic images. Our model, which we dub DP-SIMS, achieves state-of-the-art results in terms of image quality and consistency with the input label maps on ADE-20K, COCO-Stuff, and Cityscapes.
  arXiv  Detail & Related papers  (2023-12-20T09:39:19Z)
• Towards Pragmatic Semantic Image Synthesis for Urban Scenes [4.36080478413575]
  We present a new task: given a dataset with synthetic images and labels and a dataset with unlabeled real images, our goal is to learn a model that can generate images with the content of the input mask and the appearance of real images. We leverage the synthetic image as a guide to the content of the generated image by penalizing the difference between their high-level features on a patch level. In contrast to previous works which employ one discriminator that overfits the target domain semantic distribution, we employ a discriminator for the whole image and multiscale discriminators on the image patches.
  arXiv  Detail & Related papers  (2023-05-16T18:01:12Z)
• Semantic Image Synthesis via Diffusion Models [159.4285444680301]
  Denoising Diffusion Probabilistic Models (DDPMs) have achieved remarkable success in various image generation tasks. Recent work on semantic image synthesis mainly follows the emphde facto Generative Adversarial Nets (GANs)
  arXiv  Detail & Related papers  (2022-06-30T18:31:51Z)
• IR-GAN: Image Manipulation with Linguistic Instruction by Increment Reasoning [110.7118381246156]
  Increment Reasoning Generative Adversarial Network (IR-GAN) aims to reason consistency between visual increment in images and semantic increment in instructions. First, we introduce the word-level and instruction-level instruction encoders to learn user's intention from history-correlated instructions as semantic increment. Second, we embed the representation of semantic increment into that of source image for generating target image, where source image plays the role of referring auxiliary.
  arXiv  Detail & Related papers  (2022-04-02T07:48:39Z)
• USIS: Unsupervised Semantic Image Synthesis [9.613134538472801]
  We propose a new Unsupervised paradigm for Semantic Image Synthesis (USIS) USIS learns to output images with visually separable semantic classes using a self-supervised segmentation loss. In order to match the color and texture distribution of real images without losing high-frequency information, we propose to use whole image wavelet-based discrimination.
  arXiv  Detail & Related papers  (2021-09-29T20:48:41Z)
• Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization [112.68171734288237]
  We propose a novel framework for discriminative pixel-level tasks using a generative model of both images and labels. We learn a generative adversarial network that captures the joint image-label distribution and is trained efficiently using a large set of unlabeled images. We demonstrate strong in-domain performance compared to several baselines, and are the first to showcase extreme out-of-domain generalization.
  arXiv  Detail & Related papers  (2021-04-12T21:41:25Z)
• Group-Wise Semantic Mining for Weakly Supervised Semantic Segmentation [49.90178055521207]
  This work addresses weakly supervised semantic segmentation (WSSS), with the goal of bridging the gap between image-level annotations and pixel-level segmentation. We formulate WSSS as a novel group-wise learning task that explicitly models semantic dependencies in a group of images to estimate more reliable pseudo ground-truths. In particular, we devise a graph neural network (GNN) for group-wise semantic mining, wherein input images are represented as graph nodes.
  arXiv  Detail & Related papers  (2020-12-09T12:40:13Z)
• You Only Need Adversarial Supervision for Semantic Image Synthesis [84.83711654797342]
  We propose a novel, simplified GAN model, which needs only adversarial supervision to achieve high quality results. We show that images synthesized by our model are more diverse and follow the color and texture of real images more closely.
  arXiv  Detail & Related papers  (2020-12-08T23:00:48Z)
• Semantically Adaptive Image-to-image Translation for Domain Adaptation of Semantic Segmentation [1.8275108630751844]
  We address the problem of domain adaptation for semantic segmentation of street scenes. Many state-of-the-art approaches focus on translating the source image while imposing that the result should be semantically consistent with the input. We advocate that the image semantics can also be exploited to guide the translation algorithm.
  arXiv  Detail & Related papers  (2020-09-02T16:16:50Z)
• Controllable Image Synthesis via SegVAE [89.04391680233493]
  A semantic map is commonly used intermediate representation for conditional image generation. In this work, we specifically target at generating semantic maps given a label-set consisting of desired categories. The proposed framework, SegVAE, synthesizes semantic maps in an iterative manner using conditional variational autoencoder.
  arXiv  Detail & Related papers  (2020-07-16T15:18:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.