Learning Disentangled Avatars with Hybrid 3D Representations

• URL: http://arxiv.org/abs/2309.06441v1
• Date: Tue, 12 Sep 2023 17:59:36 GMT
• Title: Learning Disentangled Avatars with Hybrid 3D Representations
• Authors: Yao Feng, Weiyang Liu, Timo Bolkart, Jinlong Yang, Marc Pollefeys, Michael J. Black
• Abstract summary: We present Disentangled Avatars(DELTA) which models humans with hybrid explicit-implicit 3D representations. We consider the disentanglement of the human body and clothing and in the second, we disentangle the face and hair. We show how these two applications can be easily combined to model full-body avatars.
• Score: 102.9632315060652
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Tremendous efforts have been made to learn animatable and photorealistic human avatars. Towards this end, both explicit and implicit 3D representations are heavily studied for a holistic modeling and capture of the whole human (e.g., body, clothing, face and hair), but neither representation is an optimal choice in terms of representation efficacy since different parts of the human avatar have different modeling desiderata. For example, meshes are generally not suitable for modeling clothing and hair. Motivated by this, we present Disentangled Avatars~(DELTA), which models humans with hybrid explicit-implicit 3D representations. DELTA takes a monocular RGB video as input, and produces a human avatar with separate body and clothing/hair layers. Specifically, we demonstrate two important applications for DELTA. For the first one, we consider the disentanglement of the human body and clothing and in the second, we disentangle the face and hair. To do so, DELTA represents the body or face with an explicit mesh-based parametric 3D model and the clothing or hair with an implicit neural radiance field. To make this possible, we design an end-to-end differentiable renderer that integrates meshes into volumetric rendering, enabling DELTA to learn directly from monocular videos without any 3D supervision. Finally, we show that how these two applications can be easily combined to model full-body avatars, such that the hair, face, body and clothing can be fully disentangled yet jointly rendered. Such a disentanglement enables hair and clothing transfer to arbitrary body shapes. We empirically validate the effectiveness of DELTA's disentanglement by demonstrating its promising performance on disentangled reconstruction, virtual clothing try-on and hairstyle transfer. To facilitate future research, we also release an open-sourced pipeline for the study of hybrid human avatar modeling.

Related papers

• DEGAS: Detailed Expressions on Full-Body Gaussian Avatars [13.683836322899953]
  We present DEGAS, the first 3D Gaussian Splatting (3DGS)-based modeling method for full-body avatars with rich facial expressions. We propose to adopt the expression latent space trained solely on 2D portrait images, bridging the gap between 2D talking faces and 3D avatars.
  arXiv  Detail & Related papers  (2024-08-20T06:52:03Z)
• Text-Guided Generation and Editing of Compositional 3D Avatars [59.584042376006316]
  Our goal is to create a realistic 3D facial avatar with hair and accessories using only a text description. Existing methods either lack realism, produce unrealistic shapes, or do not support editing.
  arXiv  Detail & Related papers  (2023-09-13T17:59:56Z)
• AvatarFusion: Zero-shot Generation of Clothing-Decoupled 3D Avatars Using 2D Diffusion [34.609403685504944]
  We present AvatarFusion, a framework for zero-shot text-to-avatar generation. We use a latent diffusion model to provide pixel-level guidance for generating human-realistic avatars. We also introduce a novel optimization method, called Pixel-Semantics Difference-Sampling (PS-DS), which semantically separates the generation of body and clothes.
  arXiv  Detail & Related papers  (2023-07-13T02:19:56Z)
• Capturing and Animation of Body and Clothing from Monocular Video [105.87228128022804]
  We present SCARF, a hybrid model combining a mesh-based body with a neural radiance field. integrating the mesh into the rendering enables us to optimize SCARF directly from monocular videos. We demonstrate that SCARFs clothing with higher visual quality than existing methods, that the clothing deforms with changing body pose and body shape, and that clothing can be successfully transferred between avatars of different subjects.
  arXiv  Detail & Related papers  (2022-10-04T19:34:05Z)
• gDNA: Towards Generative Detailed Neural Avatars [94.9804106939663]
  We show that our model is able to generate natural human avatars wearing diverse and detailed clothing. Our method can be used on the task of fitting human models to raw scans, outperforming the previous state-of-the-art.
  arXiv  Detail & Related papers  (2022-01-11T18:46:38Z)
• ICON: Implicit Clothed humans Obtained from Normals [49.5397825300977]
  Implicit functions are well suited to the first task, as they can capture details like hair or clothes. ICON infers detailed clothed-human normals conditioned on the SMPL(-X) normals. ICON takes a step towards robust 3D clothed human reconstruction from in-the-wild images.
  arXiv  Detail & Related papers  (2021-12-16T18:59:41Z)
• The Power of Points for Modeling Humans in Clothing [60.00557674969284]
  Currently it requires an artist to create 3D human avatars with realistic clothing that can move naturally. We show that a 3D representation can capture varied topology at high resolution and that can be learned from data. We train a neural network with a novel local clothing geometric feature to represent the shape of different outfits.
  arXiv  Detail & Related papers  (2021-09-02T17:58:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.