Multimodal Face Synthesis from Visual Attributes

• URL: http://arxiv.org/abs/2104.04362v1
• Date: Fri, 9 Apr 2021 13:47:23 GMT
• Title: Multimodal Face Synthesis from Visual Attributes
• Authors: Xing Di, Vishal M. Patel
• Abstract summary: We propose a novel generative adversarial network that simultaneously synthesizes identity preserving multimodal face images. multimodal stretch-in modules are introduced in the discriminator which discriminates between real and fake images.
• Score: 85.87796260802223
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Synthesis of face images from visual attributes is an important problem in computer vision and biometrics due to its applications in law enforcement and entertainment. Recent advances in deep generative networks have made it possible to synthesize high-quality face images from visual attributes. However, existing methods are specifically designed for generating unimodal images (i.e visible faces) from attributes. In this paper, we propose a novel generative adversarial network that simultaneously synthesizes identity preserving multimodal face images (i.e. visible, sketch, thermal, etc.) from visual attributes without requiring paired data in different domains for training the network. We introduce a novel generator with multimodal stretch-out modules to simultaneously synthesize multimodal face images. Additionally, multimodal stretch-in modules are introduced in the discriminator which discriminates between real and fake images. Extensive experiments and comparisons with several state-of-the-art methods are performed to verify the effectiveness of the proposed attribute-based multimodal synthesis method.

Related papers

• Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretraining [48.98105914356609]
  Lumina-mGPT is a family of multimodal autoregressive models capable of various vision and language tasks. We introduce Ominiponent Supervised Finetuning, transforming Lumina-mGPT into a foundation model that seamlessly achieves omnipotent task unification.
  arXiv  Detail & Related papers  (2024-08-05T17:46:53Z)
• Many-to-many Image Generation with Auto-regressive Diffusion Models [59.5041405824704]
  This paper introduces a domain-general framework for many-to-many image generation, capable of producing interrelated image series from a given set of images. We present MIS, a novel large-scale multi-image dataset, containing 12M synthetic multi-image samples, each with 25 interconnected images. We learn M2M, an autoregressive model for many-to-many generation, where each image is modeled within a diffusion framework.
  arXiv  Detail & Related papers  (2024-04-03T23:20:40Z)
• Unified Brain MR-Ultrasound Synthesis using Multi-Modal Hierarchical Representations [34.821129614819604]
  We introduce MHVAE, a deep hierarchical variational auto-encoder (VAE) that synthesizes missing images from various modalities. Extending multi-modal VAEs with a hierarchical latent structure, we introduce a probabilistic formulation for fusing multi-modal images in a common latent representation. Our model outperformed multi-modal VAEs, conditional GANs, and the current state-of-the-art unified method (ResViT) for missing images.
  arXiv  Detail & Related papers  (2023-09-15T20:21:03Z)
• Generative Adversarial Networks for Brain Images Synthesis: A Review [2.609784101826762]
  In medical imaging, image synthesis is the estimation process of one image (sequence, modality) from another image (sequence, modality) generative adversarial network (GAN) as one of the most popular generative-based deep learning methods. We summarized the recent developments of GANs for cross-modality brain image synthesis including CT to PET, CT to MRI, MRI to PET, and vice versa.
  arXiv  Detail & Related papers  (2023-05-16T17:28:06Z)
• Hierarchical Forgery Classifier On Multi-modality Face Forgery Clues [61.37306431455152]
  We propose a novel Hierarchical Forgery for Multi-modality Face Forgery Detection (HFC-MFFD) The HFC-MFFD learns robust patches-based hybrid representation to enhance forgery authentication in multiple-modality scenarios. The specific hierarchical face forgery is proposed to alleviate the class imbalance problem and further boost detection performance.
  arXiv  Detail & Related papers  (2022-12-30T10:54:29Z)
• Diversifying Semantic Image Synthesis and Editing via Class- and Layer-wise VAEs [8.528384027684192]
  We propose a class- and layer-wise extension to the variational autoencoder framework that allows flexible control over each object class at the local to global levels. We demonstrate that our method generates images that are both plausible and more diverse compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-06-25T04:12:05Z)
• Quality Guided Sketch-to-Photo Image Synthesis [12.617078020344618]
  We propose a generative adversarial network that synthesizes a single sketch into multiple synthetic images with unique attributes like hair color, sex, etc. Our approach is aimed at improving the visual appeal of the synthesised images while incorporating multiple attribute assignment to the generator without compromising the identity of the synthesised image.
  arXiv  Detail & Related papers  (2020-04-20T16:00:01Z)
• Multimodal Image Synthesis with Conditional Implicit Maximum Likelihood Estimation [54.17177006826262]
  We develop a new generic conditional image synthesis method based on Implicit Maximum Likelihood Estimation (IMLE) We demonstrate improved multimodal image synthesis performance on two tasks, single image super-resolution and image synthesis from scene layouts.
  arXiv  Detail & Related papers  (2020-04-07T03:06:55Z)
• Hi-Net: Hybrid-fusion Network for Multi-modal MR Image Synthesis [143.55901940771568]
  We propose a novel Hybrid-fusion Network (Hi-Net) for multi-modal MR image synthesis. In our Hi-Net, a modality-specific network is utilized to learn representations for each individual modality. A multi-modal synthesis network is designed to densely combine the latent representation with hierarchical features from each modality.
  arXiv  Detail & Related papers  (2020-02-11T08:26:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.