Topology-aware Human Avatars with Semantically-guided Gaussian Splatting

• URL: http://arxiv.org/abs/2408.09665v2
• Date: Tue, 19 Nov 2024 12:48:43 GMT
• Title: Topology-aware Human Avatars with Semantically-guided Gaussian Splatting
• Authors: Haoyu Zhao, Chen Yang, Hao Wang, Xingyue Zhao, Wei Shen,
• Abstract summary: We propose SG-GS, which uses semantics-embedded 3D Gaussians, skeleton-driven rigid deformation, and non-rigid cloth dynamics deformation to create photo-realistic human avatars. We employ a 3D network that integrates both topological and geometric associations for human avatar deformation.
• Score: 18.421585526595944
• License:
• Abstract: Reconstructing photo-realistic and topology-aware animatable human avatars from monocular videos remains challenging in computer vision and graphics. Recently, methods using 3D Gaussians to represent the human body have emerged, offering faster optimization and real-time rendering. However, due to ignoring the crucial role of human body semantic information which represents the explicit topological and intrinsic structure within human body, they fail to achieve fine-detail reconstruction of human avatars. To address this issue, we propose SG-GS, which uses semantics-embedded 3D Gaussians, skeleton-driven rigid deformation, and non-rigid cloth dynamics deformation to create photo-realistic human avatars. We then design a Semantic Human-Body Annotator (SHA) which utilizes SMPL's semantic prior for efficient body part semantic labeling. The generated labels are used to guide the optimization of semantic attributes of Gaussian. To capture the explicit topological structure of the human body, we employ a 3D network that integrates both topological and geometric associations for human avatar deformation. We further implement three key strategies to enhance the semantic accuracy of 3D Gaussians and rendering quality: semantic projection with 2D regularization, semantic-guided density regularization and semantic-aware regularization with neighborhood consistency. Extensive experiments demonstrate that SG-GS achieves state-of-the-art geometry and appearance reconstruction performance.

Related papers

• Large Spatial Model: End-to-end Unposed Images to Semantic 3D [79.94479633598102]
  Large Spatial Model (LSM) processes unposed RGB images directly into semantic radiance fields. LSM simultaneously estimates geometry, appearance, and semantics in a single feed-forward operation. It can generate versatile label maps by interacting with language at novel viewpoints.
  arXiv  Detail & Related papers  (2024-10-24T17:54:42Z)
• HumanSplat: Generalizable Single-Image Human Gaussian Splatting with Structure Priors [47.62426718293504]
  HumanSplat predicts the 3D Gaussian Splatting properties of any human from a single input image. HumanSplat surpasses existing state-of-the-art methods in achieving photorealistic novel-view synthesis.
  arXiv  Detail & Related papers  (2024-06-18T10:05:33Z)
• Gaussian Control with Hierarchical Semantic Graphs in 3D Human Recovery [15.58274601909995]
  We introduce the Hierarchical Graph Human Gaussian Control (HUGS) framework for achieving high-fidelity 3D human reconstruction. Our approach involves leveraging explicitly semantic priors of body parts to ensure the consistency of geometric topology. Our method exhibits superior performance in human body reconstruction, particularly in enhancing surface details and accurately reconstructing body part junctions.
  arXiv  Detail & Related papers  (2024-05-21T03:40:56Z)
• Deformable 3D Gaussian Splatting for Animatable Human Avatars [50.61374254699761]
  We propose a fully explicit approach to construct a digital avatar from as little as a single monocular sequence. ParDy-Human constitutes an explicit model for realistic dynamic human avatars which requires significantly fewer training views and images. Our avatars learning is free of additional annotations such as Splat masks and can be trained with variable backgrounds while inferring full-resolution images efficiently even on consumer hardware.
  arXiv  Detail & Related papers  (2023-12-22T20:56:46Z)
• HyperHuman: Hyper-Realistic Human Generation with Latent Structural Diffusion [114.15397904945185]
  We propose a unified framework, HyperHuman, that generates in-the-wild human images of high realism and diverse layouts. Our model enforces the joint learning of image appearance, spatial relationship, and geometry in a unified network. Our framework yields the state-of-the-art performance, generating hyper-realistic human images under diverse scenarios.
  arXiv  Detail & Related papers  (2023-10-12T17:59:34Z)
• AvatarGen: A 3D Generative Model for Animatable Human Avatars [108.11137221845352]
  AvatarGen is an unsupervised generation of 3D-aware clothed humans with various appearances and controllable geometries. Our method can generate animatable 3D human avatars with high-quality appearance and geometry modeling. It is competent for many applications, e.g., single-view reconstruction, re-animation, and text-guided synthesis/editing.
  arXiv  Detail & Related papers  (2022-11-26T15:15:45Z)
• 3D-Aware Semantic-Guided Generative Model for Human Synthesis [67.86621343494998]
  This paper proposes a 3D-aware Semantic-Guided Generative Model (3D-SGAN) for human image synthesis. Our experiments on the DeepFashion dataset show that 3D-SGAN significantly outperforms the most recent baselines.
  arXiv  Detail & Related papers  (2021-12-02T17:10:53Z)
• THUNDR: Transformer-based 3D HUmaN Reconstruction with Markers [67.8628917474705]
  THUNDR is a transformer-based deep neural network methodology to reconstruct the 3d pose and shape of people. We show state-of-the-art results on Human3.6M and 3DPW, for both the fully-supervised and the self-supervised models. We observe very solid 3d reconstruction performance for difficult human poses collected in the wild.
  arXiv  Detail & Related papers  (2021-06-17T09:09:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.