SplattingAvatar: Realistic Real-Time Human Avatars with Mesh-Embedded Gaussian Splatting

• URL: http://arxiv.org/abs/2403.05087v1
• Date: Fri, 8 Mar 2024 06:28:09 GMT
• Title: SplattingAvatar: Realistic Real-Time Human Avatars with Mesh-Embedded Gaussian Splatting
• Authors: Zhijing Shao, Zhaolong Wang, Zhuang Li, Duotun Wang, Xiangru Lin, Yu Zhang, Mingming Fan, Zeyu Wang
• Abstract summary: We present SplattingAvatar, a hybrid 3D representation of human avatars with Gaussian Splatting embedded on a triangle mesh. SplattingAvatar renders over 300 FPS on a modern GPU and 30 FPS on a mobile device.
• Score: 26.849406891462557
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: We present SplattingAvatar, a hybrid 3D representation of photorealistic human avatars with Gaussian Splatting embedded on a triangle mesh, which renders over 300 FPS on a modern GPU and 30 FPS on a mobile device. We disentangle the motion and appearance of a virtual human with explicit mesh geometry and implicit appearance modeling with Gaussian Splatting. The Gaussians are defined by barycentric coordinates and displacement on a triangle mesh as Phong surfaces. We extend lifted optimization to simultaneously optimize the parameters of the Gaussians while walking on the triangle mesh. SplattingAvatar is a hybrid representation of virtual humans where the mesh represents low-frequency motion and surface deformation, while the Gaussians take over the high-frequency geometry and detailed appearance. Unlike existing deformation methods that rely on an MLP-based linear blend skinning (LBS) field for motion, we control the rotation and translation of the Gaussians directly by mesh, which empowers its compatibility with various animation techniques, e.g., skeletal animation, blend shapes, and mesh editing. Trainable from monocular videos for both full-body and head avatars, SplattingAvatar shows state-of-the-art rendering quality across multiple datasets.

Related papers

• RMAvatar: Photorealistic Human Avatar Reconstruction from Monocular Video Based on Rectified Mesh-embedded Gaussians [17.99475480217687]
  We introduce RMAvatar, a novel human avatar representation with Gaussian splatting embedded on mesh to learn avatar from a monocular video. We utilize the explicit mesh geometry to represent motion and shape of a virtual human and implicit appearance rendering with Gaussian Splatting. We conduct extensive experiments on public datasets, RMAvatar shows state-of-the-art performance on both rendering quality and quantitative evaluations.
  arXiv  Detail & Related papers  (2025-01-13T07:32:44Z)
• MeshGS: Adaptive Mesh-Aligned Gaussian Splatting for High-Quality Rendering [61.64903786502728]
  We propose a novel approach that integrates mesh representation with 3D Gaussian splats to perform high-quality rendering of reconstructed real-world scenes. We consider the distance between each Gaussian splat and the mesh surface to distinguish between tightly-bound and loosely-bound splats. Our method surpasses recent mesh-based neural rendering techniques by achieving a 2dB higher PSNR, and outperforms mesh-based Gaussian splatting methods by 1.3 dB PSNR.
  arXiv  Detail & Related papers  (2024-10-11T16:07:59Z)
• DreamMesh4D: Video-to-4D Generation with Sparse-Controlled Gaussian-Mesh Hybrid Representation [10.250715657201363]
  We introduce DreamMesh4D, a novel framework combining mesh representation with geometric skinning technique to generate high-quality 4D object from a monocular video. Our method is compatible with modern graphic pipelines, showcasing its potential in the 3D gaming and film industry.
  arXiv  Detail & Related papers  (2024-10-09T10:41:08Z)
• 3D Gaussian Blendshapes for Head Avatar Animation [31.488663463060416]
  We introduce 3D Gaussian blendshapes for modeling photorealistic head avatars. The avatar model of an arbitrary expression can be effectively generated by combining the neutral model and expression blendshapes. High-fidelity head avatar animations can be synthesized in real time using Gaussian splatting.
  arXiv  Detail & Related papers  (2024-04-30T09:45:41Z)
• Hybrid Explicit Representation for Ultra-Realistic Head Avatars [55.829497543262214]
  We introduce a novel approach to creating ultra-realistic head avatars and rendering them in real-time. UV-mapped 3D mesh is utilized to capture sharp and rich textures on smooth surfaces, while 3D Gaussian Splatting is employed to represent complex geometric structures. Experiments that our modeled results exceed those of state-of-the-art approaches.
  arXiv  Detail & Related papers  (2024-03-18T04:01:26Z)
• Deformable 3D Gaussian Splatting for Animatable Human Avatars [50.61374254699761]
  We propose a fully explicit approach to construct a digital avatar from as little as a single monocular sequence. ParDy-Human constitutes an explicit model for realistic dynamic human avatars which requires significantly fewer training views and images. Our avatars learning is free of additional annotations such as Splat masks and can be trained with variable backgrounds while inferring full-resolution images efficiently even on consumer hardware.
  arXiv  Detail & Related papers  (2023-12-22T20:56:46Z)
• GAvatar: Animatable 3D Gaussian Avatars with Implicit Mesh Learning [60.33970027554299]
  Gaussian splatting has emerged as a powerful 3D representation that harnesses the advantages of both explicit (mesh) and implicit (NeRF) 3D representations. In this paper, we seek to leverage Gaussian splatting to generate realistic animatable avatars from textual descriptions. Our proposed method, GAvatar, enables the large-scale generation of diverse animatable avatars using only text prompts.
  arXiv  Detail & Related papers  (2023-12-18T18:59:12Z)
• GaussianAvatar: Towards Realistic Human Avatar Modeling from a Single Video via Animatable 3D Gaussians [51.46168990249278]
  We present an efficient approach to creating realistic human avatars with dynamic 3D appearances from a single video. GustafAvatar is validated on both the public dataset and our collected dataset.
  arXiv  Detail & Related papers  (2023-12-04T18:55:45Z)
• GaussianAvatars: Photorealistic Head Avatars with Rigged 3D Gaussians [41.378083782290545]
  We introduce a new method to create photorealistic head avatars that are fully controllable in terms of expression, pose, and viewpoint. The core idea is a dynamic 3D representation based on 3D Gaussian splats rigged to a parametric morphable face model. We demonstrate the animation capabilities of our photorealistic avatar in several challenging scenarios.
  arXiv  Detail & Related papers  (2023-12-04T17:28:35Z)
• FLARE: Fast Learning of Animatable and Relightable Mesh Avatars [64.48254296523977]
  Our goal is to efficiently learn personalized animatable 3D head avatars from videos that are geometrically accurate, realistic, relightable, and compatible with current rendering systems. We introduce FLARE, a technique that enables the creation of animatable and relightable avatars from a single monocular video.
  arXiv  Detail & Related papers  (2023-10-26T16:13:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.