Expressive Whole-Body 3D Gaussian Avatar

• URL: http://arxiv.org/abs/2407.21686v1
• Date: Wed, 31 Jul 2024 15:29:13 GMT
• Title: Expressive Whole-Body 3D Gaussian Avatar
• Authors: Gyeongsik Moon, Takaaki Shiratori, Shunsuke Saito,
• Abstract summary: We present ExAvatar, an expressive whole-body 3D human avatar learned from a short monocular video. The main challenges are 1) a limited diversity of facial expressions and poses in the video and 2) the absence of 3D observations, such as 3D scans and RGBD images.
• Score: 34.3179424934446
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Facial expression and hand motions are necessary to express our emotions and interact with the world. Nevertheless, most of the 3D human avatars modeled from a casually captured video only support body motions without facial expressions and hand motions.In this work, we present ExAvatar, an expressive whole-body 3D human avatar learned from a short monocular video. We design ExAvatar as a combination of the whole-body parametric mesh model (SMPL-X) and 3D Gaussian Splatting (3DGS). The main challenges are 1) a limited diversity of facial expressions and poses in the video and 2) the absence of 3D observations, such as 3D scans and RGBD images. The limited diversity in the video makes animations with novel facial expressions and poses non-trivial. In addition, the absence of 3D observations could cause significant ambiguity in human parts that are not observed in the video, which can result in noticeable artifacts under novel motions. To address them, we introduce our hybrid representation of the mesh and 3D Gaussians. Our hybrid representation treats each 3D Gaussian as a vertex on the surface with pre-defined connectivity information (i.e., triangle faces) between them following the mesh topology of SMPL-X. It makes our ExAvatar animatable with novel facial expressions by driven by the facial expression space of SMPL-X. In addition, by using connectivity-based regularizers, we significantly reduce artifacts in novel facial expressions and poses.

Related papers

• DreamWaltz-G: Expressive 3D Gaussian Avatars from Skeleton-Guided 2D Diffusion [69.67970568012599]
  We present DreamWaltz-G, a novel learning framework for animatable 3D avatar generation from text. The core of this framework lies in Score Distillation and Hybrid 3D Gaussian Avatar representation. Our framework further supports diverse applications, including human video reenactment and multi-subject scene composition.
  arXiv  Detail & Related papers  (2024-09-25T17:59:45Z)
• DEGAS: Detailed Expressions on Full-Body Gaussian Avatars [13.683836322899953]
  We present DEGAS, the first 3D Gaussian Splatting (3DGS)-based modeling method for full-body avatars with rich facial expressions. We propose to adopt the expression latent space trained solely on 2D portrait images, bridging the gap between 2D talking faces and 3D avatars.
  arXiv  Detail & Related papers  (2024-08-20T06:52:03Z)
• iHuman: Instant Animatable Digital Humans From Monocular Videos [16.98924995658091]
  We present a fast, simple, yet effective method for creating animatable 3D digital humans from monocular videos. This work achieves and illustrates the need of accurate 3D mesh-type modelling of the human body. Our method is faster by an order of magnitude (in terms of training time) than its closest competitor.
  arXiv  Detail & Related papers  (2024-07-15T18:51:51Z)
• Deformable 3D Gaussian Splatting for Animatable Human Avatars [50.61374254699761]
  We propose a fully explicit approach to construct a digital avatar from as little as a single monocular sequence. ParDy-Human constitutes an explicit model for realistic dynamic human avatars which requires significantly fewer training views and images. Our avatars learning is free of additional annotations such as Splat masks and can be trained with variable backgrounds while inferring full-resolution images efficiently even on consumer hardware.
  arXiv  Detail & Related papers  (2023-12-22T20:56:46Z)
• Reality's Canvas, Language's Brush: Crafting 3D Avatars from Monocular Video [14.140380599168628]
  ReCaLaB is a pipeline that learns high-fidelity 3D human avatars from just a single RGB video. A pose-conditioned NeRF is optimized to volumetrically represent a human subject in canonical T-pose. An image-conditioned diffusion model thereby helps to animate appearance and pose of the 3D avatar to create video sequences with previously unseen human motion.
  arXiv  Detail & Related papers  (2023-12-08T01:53:06Z)
• DF-3DFace: One-to-Many Speech Synchronized 3D Face Animation with Diffusion [68.85904927374165]
  We propose DF-3DFace, a diffusion-driven speech-to-3D face mesh synthesis. It captures the complex one-to-many relationships between speech and 3D face based on diffusion. It simultaneously achieves more realistic facial animation than the state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-08-23T04:14:55Z)
• AniFaceGAN: Animatable 3D-Aware Face Image Generation for Video Avatars [71.00322191446203]
  2D generative models often suffer from undesirable artifacts when rendering images from different camera viewpoints. Recently, 3D-aware GANs extend 2D GANs for explicit disentanglement of camera pose by leveraging 3D scene representations. We propose an animatable 3D-aware GAN for multiview consistent face animation generation.
  arXiv  Detail & Related papers  (2022-10-12T17:59:56Z)
• ICON: Implicit Clothed humans Obtained from Normals [49.5397825300977]
  Implicit functions are well suited to the first task, as they can capture details like hair or clothes. ICON infers detailed clothed-human normals conditioned on the SMPL(-X) normals. ICON takes a step towards robust 3D clothed human reconstruction from in-the-wild images.
  arXiv  Detail & Related papers  (2021-12-16T18:59:41Z)
• 3D-TalkEmo: Learning to Synthesize 3D Emotional Talking Head [13.305263646852087]
  We introduce 3D-TalkEmo, a deep neural network that generates 3D talking head animation with various emotions. We also create a large 3D dataset with synchronized audios and videos, rich corpus, as well as various emotion states of different persons.
  arXiv  Detail & Related papers  (2021-04-25T02:48:19Z)
• Audio- and Gaze-driven Facial Animation of Codec Avatars [149.0094713268313]
  We describe the first approach to animate Codec Avatars in real-time using audio and/or eye tracking. Our goal is to display expressive conversations between individuals that exhibit important social signals.
  arXiv  Detail & Related papers  (2020-08-11T22:28:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.