SplattingAvatar: Realistic Real-Time Human Avatars with Mesh-Embedded
Gaussian Splatting
- URL: http://arxiv.org/abs/2403.05087v1
- Date: Fri, 8 Mar 2024 06:28:09 GMT
- Title: SplattingAvatar: Realistic Real-Time Human Avatars with Mesh-Embedded
Gaussian Splatting
- Authors: Zhijing Shao, Zhaolong Wang, Zhuang Li, Duotun Wang, Xiangru Lin, Yu
Zhang, Mingming Fan, Zeyu Wang
- Abstract summary: We present SplattingAvatar, a hybrid 3D representation of human avatars with Gaussian Splatting embedded on a triangle mesh.
SplattingAvatar renders over 300 FPS on a modern GPU and 30 FPS on a mobile device.
- Score: 26.849406891462557
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: We present SplattingAvatar, a hybrid 3D representation of photorealistic
human avatars with Gaussian Splatting embedded on a triangle mesh, which
renders over 300 FPS on a modern GPU and 30 FPS on a mobile device. We
disentangle the motion and appearance of a virtual human with explicit mesh
geometry and implicit appearance modeling with Gaussian Splatting. The
Gaussians are defined by barycentric coordinates and displacement on a triangle
mesh as Phong surfaces. We extend lifted optimization to simultaneously
optimize the parameters of the Gaussians while walking on the triangle mesh.
SplattingAvatar is a hybrid representation of virtual humans where the mesh
represents low-frequency motion and surface deformation, while the Gaussians
take over the high-frequency geometry and detailed appearance. Unlike existing
deformation methods that rely on an MLP-based linear blend skinning (LBS) field
for motion, we control the rotation and translation of the Gaussians directly
by mesh, which empowers its compatibility with various animation techniques,
e.g., skeletal animation, blend shapes, and mesh editing. Trainable from
monocular videos for both full-body and head avatars, SplattingAvatar shows
state-of-the-art rendering quality across multiple datasets.
Related papers
- MeshGS: Adaptive Mesh-Aligned Gaussian Splatting for High-Quality Rendering [61.64903786502728]
We propose a novel approach that integrates mesh representation with 3D Gaussian splats to perform high-quality rendering of reconstructed real-world scenes.
We consider the distance between each Gaussian splat and the mesh surface to distinguish between tightly-bound and loosely-bound splats.
Our method surpasses recent mesh-based neural rendering techniques by achieving a 2dB higher PSNR, and outperforms mesh-based Gaussian splatting methods by 1.3 dB PSNR.
arXiv Detail & Related papers (2024-10-11T16:07:59Z) - DreamMesh4D: Video-to-4D Generation with Sparse-Controlled Gaussian-Mesh Hybrid Representation [10.250715657201363]
We introduce DreamMesh4D, a novel framework combining mesh representation with geometric skinning technique to generate high-quality 4D object from a monocular video.
Our method is compatible with modern graphic pipelines, showcasing its potential in the 3D gaming and film industry.
arXiv Detail & Related papers (2024-10-09T10:41:08Z) - 3D Gaussian Blendshapes for Head Avatar Animation [31.488663463060416]
We introduce 3D Gaussian blendshapes for modeling photorealistic head avatars.
The avatar model of an arbitrary expression can be effectively generated by combining the neutral model and expression blendshapes.
High-fidelity head avatar animations can be synthesized in real time using Gaussian splatting.
arXiv Detail & Related papers (2024-04-30T09:45:41Z) - Bridging 3D Gaussian and Mesh for Freeview Video Rendering [57.21847030980905]
GauMesh bridges the 3D Gaussian and Mesh for modeling and rendering the dynamic scenes.
We show that our approach adapts the appropriate type of primitives to represent the different parts of the dynamic scene.
arXiv Detail & Related papers (2024-03-18T04:01:26Z) - PSAvatar: A Point-based Shape Model for Real-Time Head Avatar Animation with 3D Gaussian Splatting [17.78639236586134]
PSAvatar is a novel framework for animatable head avatar creation.
It employs 3D Gaussian for fine detail representation and high fidelity rendering.
We show that PSAvatar can reconstruct high-fidelity head avatars of a variety of subjects and the avatars can be animated in real-time.
arXiv Detail & Related papers (2024-01-23T16:40:47Z) - Deformable 3D Gaussian Splatting for Animatable Human Avatars [50.61374254699761]
We propose a fully explicit approach to construct a digital avatar from as little as a single monocular sequence.
ParDy-Human constitutes an explicit model for realistic dynamic human avatars which requires significantly fewer training views and images.
Our avatars learning is free of additional annotations such as Splat masks and can be trained with variable backgrounds while inferring full-resolution images efficiently even on consumer hardware.
arXiv Detail & Related papers (2023-12-22T20:56:46Z) - GAvatar: Animatable 3D Gaussian Avatars with Implicit Mesh Learning [60.33970027554299]
Gaussian splatting has emerged as a powerful 3D representation that harnesses the advantages of both explicit (mesh) and implicit (NeRF) 3D representations.
In this paper, we seek to leverage Gaussian splatting to generate realistic animatable avatars from textual descriptions.
Our proposed method, GAvatar, enables the large-scale generation of diverse animatable avatars using only text prompts.
arXiv Detail & Related papers (2023-12-18T18:59:12Z) - GaussianAvatar: Towards Realistic Human Avatar Modeling from a Single Video via Animatable 3D Gaussians [51.46168990249278]
We present an efficient approach to creating realistic human avatars with dynamic 3D appearances from a single video.
GustafAvatar is validated on both the public dataset and our collected dataset.
arXiv Detail & Related papers (2023-12-04T18:55:45Z) - GaussianAvatars: Photorealistic Head Avatars with Rigged 3D Gaussians [41.378083782290545]
We introduce a new method to create photorealistic head avatars that are fully controllable in terms of expression, pose, and viewpoint.
The core idea is a dynamic 3D representation based on 3D Gaussian splats rigged to a parametric morphable face model.
We demonstrate the animation capabilities of our photorealistic avatar in several challenging scenarios.
arXiv Detail & Related papers (2023-12-04T17:28:35Z) - FLARE: Fast Learning of Animatable and Relightable Mesh Avatars [64.48254296523977]
Our goal is to efficiently learn personalized animatable 3D head avatars from videos that are geometrically accurate, realistic, relightable, and compatible with current rendering systems.
We introduce FLARE, a technique that enables the creation of animatable and relightable avatars from a single monocular video.
arXiv Detail & Related papers (2023-10-26T16:13:00Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.