Layered 3D Human Generation via Semantic-Aware Diffusion Model

• URL: http://arxiv.org/abs/2312.05804v3
• Date: Sun, 21 Jul 2024 06:09:05 GMT
• Title: Layered 3D Human Generation via Semantic-Aware Diffusion Model
• Authors: Yi Wang, Jian Ma, Ruizhi Shao, Qiao Feng, Yu-Kun Lai, Yebin Liu, Kun Li,
• Abstract summary: We propose a text-driven layered 3D human generation framework based on a novel semantic-aware diffusion model. To keep the generated clothing consistent with the target text, we propose a semantic-confidence strategy for clothing. To match the clothing with different body shapes, we propose a SMPL-driven implicit field deformation network.
• Score: 63.459666003261276
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The generation of 3D clothed humans has attracted increasing attention in recent years. However, existing work cannot generate layered high-quality 3D humans with consistent body structures. As a result, these methods are unable to arbitrarily and separately change and edit the body and clothing of the human. In this paper, we propose a text-driven layered 3D human generation framework based on a novel physically-decoupled semantic-aware diffusion model. To keep the generated clothing consistent with the target text, we propose a semantic-confidence strategy for clothing that can eliminate the non-clothing content generated by the model. To match the clothing with different body shapes, we propose a SMPL-driven implicit field deformation network that enables the free transfer and reuse of clothing. Besides, we introduce uniform shape priors based on the SMPL model for body and clothing, respectively, which generates more diverse 3D content without being constrained by specific templates. The experimental results demonstrate that the proposed method not only generates 3D humans with consistent body structures but also allows free editing in a layered manner. The source code will be made public.

Related papers

• HumanCoser: Layered 3D Human Generation via Semantic-Aware Diffusion Model [43.66218796152962]
  This paper aims to generate physically-layered 3D humans from text prompts. We propose a novel layer-wise dressed human representation based on a physically-decoupled diffusion model. To match the clothing with different body shapes, we propose an SMPL-driven implicit field network.
  arXiv  Detail & Related papers  (2024-08-21T06:00:11Z)
• TELA: Text to Layer-wise 3D Clothed Human Generation [27.93447899876341]
  This paper addresses the task of 3D clothed human generation from textural descriptions. We propose a layer-wise clothed human representation combined with a progressive optimization strategy. Our approach achieves state-of-the-art 3D clothed human generation while also supporting cloth editing applications.
  arXiv  Detail & Related papers  (2024-04-25T17:05:38Z)
• HumanLiff: Layer-wise 3D Human Generation with Diffusion Model [55.891036415316876]
  Existing 3D human generative models mainly generate a clothed 3D human as an undetectable 3D model in a single pass. We propose HumanLiff, the first layer-wise 3D human generative model with a unified diffusion process.
  arXiv  Detail & Related papers  (2023-08-18T17:59:04Z)
• Capturing and Animation of Body and Clothing from Monocular Video [105.87228128022804]
  We present SCARF, a hybrid model combining a mesh-based body with a neural radiance field. integrating the mesh into the rendering enables us to optimize SCARF directly from monocular videos. We demonstrate that SCARFs clothing with higher visual quality than existing methods, that the clothing deforms with changing body pose and body shape, and that clothing can be successfully transferred between avatars of different subjects.
  arXiv  Detail & Related papers  (2022-10-04T19:34:05Z)
• SMPLicit: Topology-aware Generative Model for Clothed People [65.84665248796615]
  We introduce SMPLicit, a novel generative model to jointly represent body pose, shape and clothing geometry. In the experimental section, we demonstrate SMPLicit can be readily used for fitting 3D scans and for 3D reconstruction in images of dressed people.
  arXiv  Detail & Related papers  (2021-03-11T18:57:03Z)
• Neural 3D Clothes Retargeting from a Single Image [91.5030622330039]
  We present a method of clothes; generating the potential poses and deformations of a given 3D clothing template model to fit onto a person in a single RGB image. The problem is fundamentally ill-posed as attaining the ground truth data is impossible, i.e. images of people wearing the different 3D clothing template model model at exact same pose. We propose a semi-supervised learning framework that validates the physical plausibility of 3D deformation by matching with the prescribed body-to-cloth contact points and clothing to fit onto the unlabeled silhouette.
  arXiv  Detail & Related papers  (2021-01-29T20:50:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.