Chupa: Carving 3D Clothed Humans from Skinned Shape Priors using 2D Diffusion Probabilistic Models

• URL: http://arxiv.org/abs/2305.11870v3
• Date: Fri, 15 Sep 2023 12:23:21 GMT
• Title: Chupa: Carving 3D Clothed Humans from Skinned Shape Priors using 2D Diffusion Probabilistic Models
• Authors: Byungjun Kim, Patrick Kwon, Kwangho Lee, Myunggi Lee, Sookwan Han, Daesik Kim, Hanbyul Joo
• Abstract summary: We propose a 3D generation pipeline that uses diffusion models to generate realistic human digital avatars. Our method, namely, Chupa, is capable of generating realistic 3D clothed humans with better perceptual quality and identity variety.
• Score: 9.479195068754507
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose a 3D generation pipeline that uses diffusion models to generate realistic human digital avatars. Due to the wide variety of human identities, poses, and stochastic details, the generation of 3D human meshes has been a challenging problem. To address this, we decompose the problem into 2D normal map generation and normal map-based 3D reconstruction. Specifically, we first simultaneously generate realistic normal maps for the front and backside of a clothed human, dubbed dual normal maps, using a pose-conditional diffusion model. For 3D reconstruction, we "carve" the prior SMPL-X mesh to a detailed 3D mesh according to the normal maps through mesh optimization. To further enhance the high-frequency details, we present a diffusion resampling scheme on both body and facial regions, thus encouraging the generation of realistic digital avatars. We also seamlessly incorporate a recent text-to-image diffusion model to support text-based human identity control. Our method, namely, Chupa, is capable of generating realistic 3D clothed humans with better perceptual quality and identity variety.

Related papers

• Human 3Diffusion: Realistic Avatar Creation via Explicit 3D Consistent Diffusion Models [29.73743772971411]
  We propose Human 3Diffusion: Realistic Avatar Creation via Explicit 3D Consistent Diffusion. Our key insight is that 2D multi-view diffusion and 3D reconstruction models provide complementary information for each other. Our proposed framework outperforms state-of-the-art methods and enables the creation of realistic avatars from a single RGB image.
  arXiv  Detail & Related papers  (2024-06-12T17:57:25Z)
• FitDiff: Robust monocular 3D facial shape and reflectance estimation using Diffusion Models [79.65289816077629]
  We present FitDiff, a diffusion-based 3D facial avatar generative model. Our model accurately generates relightable facial avatars, utilizing an identity embedding extracted from an "in-the-wild" 2D facial image. Being the first 3D LDM conditioned on face recognition embeddings, FitDiff reconstructs relightable human avatars, that can be used as-is in common rendering engines.
  arXiv  Detail & Related papers  (2023-12-07T17:35:49Z)
• GETAvatar: Generative Textured Meshes for Animatable Human Avatars [69.56959932421057]
  We study the problem of 3D-aware full-body human generation, aiming at creating animatable human avatars with high-quality geometries and textures. We propose GETAvatar, a Generative model that directly generates Explicit Textured 3D rendering for animatable human Avatar.
  arXiv  Detail & Related papers  (2023-10-04T10:30:24Z)
• AG3D: Learning to Generate 3D Avatars from 2D Image Collections [96.28021214088746]
  We propose a new adversarial generative model of realistic 3D people from 2D images. Our method captures shape and deformation of the body and loose clothing by adopting a holistic 3D generator. We experimentally find that our method outperforms previous 3D- and articulation-aware methods in terms of geometry and appearance.
  arXiv  Detail & Related papers  (2023-05-03T17:56:24Z)
• DreamAvatar: Text-and-Shape Guided 3D Human Avatar Generation via Diffusion Models [55.71306021041785]
  We present DreamAvatar, a text-and-shape guided framework for generating high-quality 3D human avatars. We leverage the SMPL model to provide shape and pose guidance for the generation. We also jointly optimize the losses computed from the full body and from the zoomed-in 3D head to alleviate the common multi-face ''Janus'' problem.
  arXiv  Detail & Related papers  (2023-04-03T12:11:51Z)
• Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion [66.26780039133122]
  This paper presents a 3D generative model that uses diffusion models to automatically generate 3D digital avatars. The memory and processing costs in 3D are prohibitive for producing the rich details required for high-quality avatars. We can generate highly detailed avatars with realistic hairstyles and facial hair like beards.
  arXiv  Detail & Related papers  (2022-12-12T18:59:40Z)
• 3D-Aware Semantic-Guided Generative Model for Human Synthesis [67.86621343494998]
  This paper proposes a 3D-aware Semantic-Guided Generative Model (3D-SGAN) for human image synthesis. Our experiments on the DeepFashion dataset show that 3D-SGAN significantly outperforms the most recent baselines.
  arXiv  Detail & Related papers  (2021-12-02T17:10:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.