RMAvatar: Photorealistic Human Avatar Reconstruction from Monocular Video Based on Rectified Mesh-embedded Gaussians

• URL: http://arxiv.org/abs/2501.07104v1
• Date: Mon, 13 Jan 2025 07:32:44 GMT
• Title: RMAvatar: Photorealistic Human Avatar Reconstruction from Monocular Video Based on Rectified Mesh-embedded Gaussians
• Authors: Sen Peng, Weixing Xie, Zilong Wang, Xiaohu Guo, Zhonggui Chen, Baorong Yang, Xiao Dong,
• Abstract summary: We introduce RMAvatar, a novel human avatar representation with Gaussian splatting embedded on mesh to learn avatar from a monocular video. We utilize the explicit mesh geometry to represent motion and shape of a virtual human and implicit appearance rendering with Gaussian Splatting. We conduct extensive experiments on public datasets, RMAvatar shows state-of-the-art performance on both rendering quality and quantitative evaluations.
• Score: 17.99475480217687
• License:
• Abstract: We introduce RMAvatar, a novel human avatar representation with Gaussian splatting embedded on mesh to learn clothed avatar from a monocular video. We utilize the explicit mesh geometry to represent motion and shape of a virtual human and implicit appearance rendering with Gaussian Splatting. Our method consists of two main modules: Gaussian initialization module and Gaussian rectification module. We embed Gaussians into triangular faces and control their motion through the mesh, which ensures low-frequency motion and surface deformation of the avatar. Due to the limitations of LBS formula, the human skeleton is hard to control complex non-rigid transformations. We then design a pose-related Gaussian rectification module to learn fine-detailed non-rigid deformations, further improving the realism and expressiveness of the avatar. We conduct extensive experiments on public datasets, RMAvatar shows state-of-the-art performance on both rendering quality and quantitative evaluations. Please see our project page at https://rm-avatar.github.io.

Related papers

• Generalizable and Animatable Gaussian Head Avatar [50.34788590904843]
  We propose Generalizable and Animatable Gaussian head Avatar (GAGAvatar) for one-shot animatable head avatar reconstruction. We generate the parameters of 3D Gaussians from a single image in a single forward pass. Our method exhibits superior performance compared to previous methods in terms of reconstruction quality and expression accuracy.
  arXiv  Detail & Related papers  (2024-10-10T14:29:00Z)
• Gaussian Eigen Models for Human Heads [28.49783203616257]
  Current personalized neural head avatars face a trade-off: lightweight models lack detail and realism, while high-quality, animatable avatars require significant computational resources. We introduce Gaussian Eigen Models (GEM), which provide high-quality, lightweight, and easily controllable head avatars.
  arXiv  Detail & Related papers  (2024-07-05T14:30:24Z)
• Expressive Gaussian Human Avatars from Monocular RGB Video [69.56388194249942]
  We introduce EVA, a drivable human model that meticulously sculpts fine details based on 3D Gaussians and SMPL-X. We highlight the critical importance of aligning the SMPL-X model with RGB frames for effective avatar learning. We propose a context-aware adaptive density control strategy, which is adaptively adjusting the gradient thresholds.
  arXiv  Detail & Related papers  (2024-07-03T15:36:27Z)
• UV Gaussians: Joint Learning of Mesh Deformation and Gaussian Textures for Human Avatar Modeling [71.87807614875497]
  We propose UV Gaussians, which models the 3D human body by jointly learning mesh deformations and 2D UV-space Gaussian textures. We collect and process a new dataset of human motion, which includes multi-view images, scanned models, parametric model registration, and corresponding texture maps. Experimental results demonstrate that our method achieves state-of-the-art synthesis of novel view and novel pose.
  arXiv  Detail & Related papers  (2024-03-18T09:03:56Z)
• SplattingAvatar: Realistic Real-Time Human Avatars with Mesh-Embedded Gaussian Splatting [26.849406891462557]
  We present SplattingAvatar, a hybrid 3D representation of human avatars with Gaussian Splatting embedded on a triangle mesh. SplattingAvatar renders over 300 FPS on a modern GPU and 30 FPS on a mobile device.
  arXiv  Detail & Related papers  (2024-03-08T06:28:09Z)
• GVA: Reconstructing Vivid 3D Gaussian Avatars from Monocular Videos [56.40776739573832]
  We present a novel method that facilitates the creation of vivid 3D Gaussian avatars from monocular video inputs (GVA) Our innovation lies in addressing the intricate challenges of delivering high-fidelity human body reconstructions. We introduce a pose refinement technique to improve hand and foot pose accuracy by aligning normal maps and silhouettes.
  arXiv  Detail & Related papers  (2024-02-26T14:40:15Z)
• Deformable 3D Gaussian Splatting for Animatable Human Avatars [50.61374254699761]
  We propose a fully explicit approach to construct a digital avatar from as little as a single monocular sequence. ParDy-Human constitutes an explicit model for realistic dynamic human avatars which requires significantly fewer training views and images. Our avatars learning is free of additional annotations such as Splat masks and can be trained with variable backgrounds while inferring full-resolution images efficiently even on consumer hardware.
  arXiv  Detail & Related papers  (2023-12-22T20:56:46Z)
• GAvatar: Animatable 3D Gaussian Avatars with Implicit Mesh Learning [60.33970027554299]
  Gaussian splatting has emerged as a powerful 3D representation that harnesses the advantages of both explicit (mesh) and implicit (NeRF) 3D representations. In this paper, we seek to leverage Gaussian splatting to generate realistic animatable avatars from textual descriptions. Our proposed method, GAvatar, enables the large-scale generation of diverse animatable avatars using only text prompts.
  arXiv  Detail & Related papers  (2023-12-18T18:59:12Z)
• MonoGaussianAvatar: Monocular Gaussian Point-based Head Avatar [44.125711148560605]
  MonoGaussianAvatar is a novel approach that harnesses 3D Gaussian point representation and a Gaussian deformation field to learn explicit head avatars from monocular portrait videos. Experiments demonstrate the superior performance of our method, which achieves state-of-the-art results among previous methods.
  arXiv  Detail & Related papers  (2023-12-07T18:59:31Z)
• GaussianAvatars: Photorealistic Head Avatars with Rigged 3D Gaussians [41.378083782290545]
  We introduce a new method to create photorealistic head avatars that are fully controllable in terms of expression, pose, and viewpoint. The core idea is a dynamic 3D representation based on 3D Gaussian splats rigged to a parametric morphable face model. We demonstrate the animation capabilities of our photorealistic avatar in several challenging scenarios.
  arXiv  Detail & Related papers  (2023-12-04T17:28:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.