DRDM: A Disentangled Representations Diffusion Model for Synthesizing Realistic Person Images

• URL: http://arxiv.org/abs/2412.18797v1
• Date: Wed, 25 Dec 2024 06:36:24 GMT
• Title: DRDM: A Disentangled Representations Diffusion Model for Synthesizing Realistic Person Images
• Authors: Enbo Huang, Yuan Zhang, Faliang Huang, Guangyu Zhang, Yang Liu,
• Abstract summary: We propose a Disentangled Representations Diffusion Model (DRDM) to generate photo-realistic images from source portraits. First, a pose encoder is responsible for encoding pose features into a high-dimensional space to guide the generation of person images. Second, a body-part subspace decoupling block (BSDB) disentangles features from the different body parts of a source figure and feeds them to the various layers of the noise prediction block.
• Score: 9.768951663960257
• License:
• Abstract: Person image synthesis with controllable body poses and appearances is an essential task owing to the practical needs in the context of virtual try-on, image editing and video production. However, existing methods face significant challenges with details missing, limbs distortion and the garment style deviation. To address these issues, we propose a Disentangled Representations Diffusion Model (DRDM) to generate photo-realistic images from source portraits in specific desired poses and appearances. First, a pose encoder is responsible for encoding pose features into a high-dimensional space to guide the generation of person images. Second, a body-part subspace decoupling block (BSDB) disentangles features from the different body parts of a source figure and feeds them to the various layers of the noise prediction block, thereby supplying the network with rich disentangled features for generating a realistic target image. Moreover, during inference, we develop a parsing map-based disentangled classifier-free guided sampling method, which amplifies the conditional signals of texture and pose. Extensive experimental results on the Deepfashion dataset demonstrate the effectiveness of our approach in achieving pose transfer and appearance control.

Related papers

• Coarse-to-Fine Latent Diffusion for Pose-Guided Person Image Synthesis [65.7968515029306]
  We propose a novel Coarse-to-Fine Latent Diffusion (CFLD) method for Pose-Guided Person Image Synthesis (PGPIS) A perception-refined decoder is designed to progressively refine a set of learnable queries and extract semantic understanding of person images as a coarse-grained prompt.
  arXiv  Detail & Related papers  (2024-02-28T06:07:07Z)
• RePoseDM: Recurrent Pose Alignment and Gradient Guidance for Pose Guided Image Synthesis [14.50214193838818]
  Pose-guided person image synthesis task requires re-rendering a reference image, which should have a photorealistic appearance and flawless pose transfer. We propose recurrent pose alignment to provide pose-aligned texture features as conditional guidance. This helps in learning plausible pose transfer trajectories that result in photorealism and undistorted texture details.
  arXiv  Detail & Related papers  (2023-10-24T15:16:19Z)
• IT3D: Improved Text-to-3D Generation with Explicit View Synthesis [71.68595192524843]
  This study presents a novel strategy that leverages explicitly synthesized multi-view images to address these issues. Our approach involves the utilization of image-to-image pipelines, empowered by LDMs, to generate posed high-quality images. For the incorporated discriminator, the synthesized multi-view images are considered real data, while the renderings of the optimized 3D models function as fake data.
  arXiv  Detail & Related papers  (2023-08-22T14:39:17Z)
• Person Image Synthesis via Denoising Diffusion Model [116.34633988927429]
  We show how denoising diffusion models can be applied for high-fidelity person image synthesis. Our results on two large-scale benchmarks and a user study demonstrate the photorealism of our proposed approach under challenging scenarios.
  arXiv  Detail & Related papers  (2022-11-22T18:59:50Z)
• Pose Guided Human Image Synthesis with Partially Decoupled GAN [25.800174118151638]
  Pose Guided Human Image Synthesis (PGHIS) is a challenging task of transforming a human image from the reference pose to a target pose. We propose a method by decoupling the human body into several parts to guide the synthesis of a realistic image of the person. In addition, we design a multi-head attention-based module for PGHIS.
  arXiv  Detail & Related papers  (2022-10-07T15:31:37Z)
• Controllable Person Image Synthesis with Spatially-Adaptive Warped Normalization [72.65828901909708]
  Controllable person image generation aims to produce realistic human images with desirable attributes. We introduce a novel Spatially-Adaptive Warped Normalization (SAWN), which integrates a learned flow-field to warp modulation parameters. We propose a novel self-training part replacement strategy to refine the pretrained model for the texture-transfer task.
  arXiv  Detail & Related papers  (2021-05-31T07:07:44Z)
• PISE: Person Image Synthesis and Editing with Decoupled GAN [64.70360318367943]
  We propose PISE, a novel two-stage generative model for Person Image Synthesis and Editing. For human pose transfer, we first synthesize a human parsing map aligned with the target pose to represent the shape of clothing. To decouple the shape and style of clothing, we propose joint global and local per-region encoding and normalization.
  arXiv  Detail & Related papers  (2021-03-06T04:32:06Z)
• Adversarial Semantic Data Augmentation for Human Pose Estimation [96.75411357541438]
  We propose Semantic Data Augmentation (SDA), a method that augments images by pasting segmented body parts with various semantic granularity. We also propose Adversarial Semantic Data Augmentation (ASDA), which exploits a generative network to dynamiclly predict tailored pasting configuration. State-of-the-art results are achieved on challenging benchmarks.
  arXiv  Detail & Related papers  (2020-08-03T07:56:04Z)
• MirrorNet: A Deep Bayesian Approach to Reflective 2D Pose Estimation from Human Images [42.27703025887059]
  The main problems with the standard supervised approach are that it often yields anatomically implausible poses. We propose a semi-supervised method that can make effective use of images with and without pose annotations. The results of experiments show that the proposed reflective architecture makes estimated poses anatomically plausible.
  arXiv  Detail & Related papers  (2020-04-08T05:02:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.