Diverse Semantic Image Synthesis via Probability Distribution Modeling

• URL: http://arxiv.org/abs/2103.06878v1
• Date: Thu, 11 Mar 2021 18:59:25 GMT
• Title: Diverse Semantic Image Synthesis via Probability Distribution Modeling
• Authors: Zhentao Tan and Menglei Chai and Dongdong Chen and Jing Liao and Qi Chu and Bin Liu and Gang Hua and Nenghai Yu
• Abstract summary: We propose a novel diverse semantic image synthesis framework. Our method can achieve superior diversity and comparable quality compared to state-of-the-art methods.
• Score: 103.88931623488088
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Semantic image synthesis, translating semantic layouts to photo-realistic images, is a one-to-many mapping problem. Though impressive progress has been recently made, diverse semantic synthesis that can efficiently produce semantic-level multimodal results, still remains a challenge. In this paper, we propose a novel diverse semantic image synthesis framework from the perspective of semantic class distributions, which naturally supports diverse generation at semantic or even instance level. We achieve this by modeling class-level conditional modulation parameters as continuous probability distributions instead of discrete values, and sampling per-instance modulation parameters through instance-adaptive stochastic sampling that is consistent across the network. Moreover, we propose prior noise remapping, through linear perturbation parameters encoded from paired references, to facilitate supervised training and exemplar-based instance style control at test time. Extensive experiments on multiple datasets show that our method can achieve superior diversity and comparable quality compared to state-of-the-art methods. Code will be available at \url{https://github.com/tzt101/INADE.git}

Related papers

• A Simple Approach to Unifying Diffusion-based Conditional Generation [63.389616350290595]
  We introduce a simple, unified framework to handle diverse conditional generation tasks. Our approach enables versatile capabilities via different inference-time sampling schemes. Our model supports additional capabilities like non-spatially aligned and coarse conditioning.
  arXiv  Detail & Related papers  (2024-10-15T09:41:43Z)
• SemFlow: Binding Semantic Segmentation and Image Synthesis via Rectified Flow [94.90853153808987]
  Semantic segmentation and semantic image synthesis are representative tasks in visual perception and generation. We propose a unified framework (SemFlow) and model them as a pair of reverse problems. Experiments show that our SemFlow achieves competitive results on semantic segmentation and semantic image synthesis tasks.
  arXiv  Detail & Related papers  (2024-05-30T17:34:40Z)
• Improving Denoising Diffusion Probabilistic Models via Exploiting Shared Representations [5.517338199249029]
  SR-DDPM is a class of generative models that produce high-quality images by reversing a noisy diffusion process. By exploiting the similarity between diverse data distributions, our method can scale to multiple tasks without compromising the image quality. We evaluate our method on standard image datasets and show that it outperforms both unconditional and conditional DDPM in terms of FID and SSIM metrics.
  arXiv  Detail & Related papers  (2023-11-27T22:30:26Z)
• Variation-Aware Semantic Image Synthesis [5.232306238197685]
  We introduce two simple methods to achieve variation-aware semantic image synthesis (VASIS) with a higher intra-class variation, semantic noise and position code. Our models generate more natural images and achieves slightly better FIDs and/or mIoUs than the counterparts.
  arXiv  Detail & Related papers  (2023-01-25T12:35:17Z)
• ContraFeat: Contrasting Deep Features for Semantic Discovery [102.4163768995288]
  StyleGAN has shown strong potential for disentangled semantic control. Existing semantic discovery methods on StyleGAN rely on manual selection of modified latent layers to obtain satisfactory manipulation results. We propose a model that automates this process and achieves state-of-the-art semantic discovery performance.
  arXiv  Detail & Related papers  (2022-12-14T15:22:13Z)
• Semantic Image Synthesis via Diffusion Models [159.4285444680301]
  Denoising Diffusion Probabilistic Models (DDPMs) have achieved remarkable success in various image generation tasks. Recent work on semantic image synthesis mainly follows the emphde facto Generative Adversarial Nets (GANs)
  arXiv  Detail & Related papers  (2022-06-30T18:31:51Z)
• More Control for Free! Image Synthesis with Semantic Diffusion Guidance [79.88929906247695]
  Controllable image synthesis models allow creation of diverse images based on text instructions or guidance from an example image. We introduce a novel unified framework for semantic diffusion guidance, which allows either language or image guidance, or both. We conduct experiments on FFHQ and LSUN datasets, and show results on fine-grained text-guided image synthesis.
  arXiv  Detail & Related papers  (2021-12-10T18:55:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.