Animatable 3D Gaussians for High-fidelity Synthesis of Human Motions
- URL: http://arxiv.org/abs/2311.13404v2
- Date: Mon, 27 Nov 2023 02:33:36 GMT
- Title: Animatable 3D Gaussians for High-fidelity Synthesis of Human Motions
- Authors: Keyang Ye, Tianjia Shao, Kun Zhou
- Abstract summary: We present a novel animatable 3D Gaussian model for rendering high-fidelity free-view human motions in real time.
Compared to existing NeRF-based methods, the model owns better capability in high-frequency details without the jittering problem across video frames.
- Score: 37.50707388577952
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We present a novel animatable 3D Gaussian model for rendering high-fidelity
free-view human motions in real time. Compared to existing NeRF-based methods,
the model owns better capability in synthesizing high-frequency details without
the jittering problem across video frames. The core of our model is a novel
augmented 3D Gaussian representation, which attaches each Gaussian with a
learnable code. The learnable code serves as a pose-dependent appearance
embedding for refining the erroneous appearance caused by geometric
transformation of Gaussians, based on which an appearance refinement model is
learned to produce residual Gaussian properties to match the appearance in
target pose. To force the Gaussians to learn the foreground human only without
background interference, we further design a novel alpha loss to explicitly
constrain the Gaussians within the human body. We also propose to jointly
optimize the human joint parameters to improve the appearance accuracy. The
animatable 3D Gaussian model can be learned with shallow MLPs, so new human
motions can be synthesized in real time (66 fps on avarage). Experiments show
that our model has superior performance over NeRF-based methods.
Related papers
- Generalizable Human Gaussians from Single-View Image [54.712838657788566]
We propose single-view generalizable Human Gaussian model (HGM), a diffusion-guided framework for 3D human modeling from a single image.
Although effective in hallucinating the unobserved views, the approach may generate unrealistic human pose and shapes due to the lack of supervision.
We validate our approach on publicly available datasets and demonstrate that it significantly surpasses state-of-the-art methods in terms of PSNR and SSIM.
arXiv Detail & Related papers (2024-06-10T06:38:11Z) - UV Gaussians: Joint Learning of Mesh Deformation and Gaussian Textures for Human Avatar Modeling [71.87807614875497]
We propose UV Gaussians, which models the 3D human body by jointly learning mesh deformations and 2D UV-space Gaussian textures.
We collect and process a new dataset of human motion, which includes multi-view images, scanned models, parametric model registration, and corresponding texture maps. Experimental results demonstrate that our method achieves state-of-the-art synthesis of novel view and novel pose.
arXiv Detail & Related papers (2024-03-18T09:03:56Z) - GaMeS: Mesh-Based Adapting and Modification of Gaussian Splatting [11.791944275269266]
We introduce the Gaussian Mesh Splatting (GaMeS) model, which allows modification of Gaussian components in a similar way as meshes.
We also define Gaussian splats solely based on their location on the mesh, allowing for automatic adjustments in position, scale, and rotation during animation.
arXiv Detail & Related papers (2024-02-02T14:50:23Z) - Deformable 3D Gaussian Splatting for Animatable Human Avatars [50.61374254699761]
We propose a fully explicit approach to construct a digital avatar from as little as a single monocular sequence.
ParDy-Human constitutes an explicit model for realistic dynamic human avatars which requires significantly fewer training views and images.
Our avatars learning is free of additional annotations such as Splat masks and can be trained with variable backgrounds while inferring full-resolution images efficiently even on consumer hardware.
arXiv Detail & Related papers (2023-12-22T20:56:46Z) - GAvatar: Animatable 3D Gaussian Avatars with Implicit Mesh Learning [60.33970027554299]
Gaussian splatting has emerged as a powerful 3D representation that harnesses the advantages of both explicit (mesh) and implicit (NeRF) 3D representations.
In this paper, we seek to leverage Gaussian splatting to generate realistic animatable avatars from textual descriptions.
Our proposed method, GAvatar, enables the large-scale generation of diverse animatable avatars using only text prompts.
arXiv Detail & Related papers (2023-12-18T18:59:12Z) - GauHuman: Articulated Gaussian Splatting from Monocular Human Videos [58.553979884950834]
GauHuman is a 3D human model with Gaussian Splatting for both fast training (1 2 minutes) and real-time rendering (up to 189 FPS)
GauHuman encodes Gaussian Splatting in the canonical space and transforms 3D Gaussians from canonical space to posed space with linear blend skinning (LBS)
Experiments on ZJU_Mocap and MonoCap datasets demonstrate that GauHuman achieves state-of-the-art performance quantitatively and qualitatively with fast training and real-time rendering speed.
arXiv Detail & Related papers (2023-12-05T18:59:14Z) - HumanGaussian: Text-Driven 3D Human Generation with Gaussian Splatting [113.37908093915837]
Existing methods optimize 3D representations like mesh or neural fields via score distillation sampling (SDS), which suffers from inadequate fine details or excessive training time.
In this paper, we propose an efficient yet effective framework, HumanGaussian, that generates high-quality 3D humans with fine-grained geometry and realistic appearance.
arXiv Detail & Related papers (2023-11-28T18:59:58Z) - Human Gaussian Splatting: Real-time Rendering of Animatable Avatars [8.719797382786464]
This work addresses the problem of real-time rendering of photorealistic human body avatars learned from multi-view videos.
We propose an animatable human model based on 3D Gaussian Splatting, that has recently emerged as a very efficient alternative to neural radiance fields.
Our method achieves 1.5 dB PSNR improvement over the state-of-the-art on THuman4 dataset while being able to render in real-time (80 fps for 512x512 resolution)
arXiv Detail & Related papers (2023-11-28T12:05:41Z) - SplatArmor: Articulated Gaussian splatting for animatable humans from
monocular RGB videos [15.74530749823217]
We propose SplatArmor, a novel approach for recovering detailed and animatable human models by armoring' a parameterized body model with 3D Gaussians.
Our approach represents the human as a set of 3D Gaussians within a canonical space, whose articulation is defined by extending the skinning of the underlying SMPL geometry.
We show compelling results on the ZJU MoCap and People Snapshot datasets, which underscore the effectiveness of our method for controllable human synthesis.
arXiv Detail & Related papers (2023-11-17T18:47:07Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.