GAvatar: Animatable 3D Gaussian Avatars with Implicit Mesh Learning

• URL: http://arxiv.org/abs/2312.11461v2
• Date: Fri, 29 Mar 2024 04:32:57 GMT
• Title: GAvatar: Animatable 3D Gaussian Avatars with Implicit Mesh Learning
• Authors: Ye Yuan, Xueting Li, Yangyi Huang, Shalini De Mello, Koki Nagano, Jan Kautz, Umar Iqbal,
• Abstract summary: Gaussian splatting has emerged as a powerful 3D representation that harnesses the advantages of both explicit (mesh) and implicit (NeRF) 3D representations. In this paper, we seek to leverage Gaussian splatting to generate realistic animatable avatars from textual descriptions. Our proposed method, GAvatar, enables the large-scale generation of diverse animatable avatars using only text prompts.
• Score: 60.33970027554299
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Gaussian splatting has emerged as a powerful 3D representation that harnesses the advantages of both explicit (mesh) and implicit (NeRF) 3D representations. In this paper, we seek to leverage Gaussian splatting to generate realistic animatable avatars from textual descriptions, addressing the limitations (e.g., flexibility and efficiency) imposed by mesh or NeRF-based representations. However, a naive application of Gaussian splatting cannot generate high-quality animatable avatars and suffers from learning instability; it also cannot capture fine avatar geometries and often leads to degenerate body parts. To tackle these problems, we first propose a primitive-based 3D Gaussian representation where Gaussians are defined inside pose-driven primitives to facilitate animation. Second, to stabilize and amortize the learning of millions of Gaussians, we propose to use neural implicit fields to predict the Gaussian attributes (e.g., colors). Finally, to capture fine avatar geometries and extract detailed meshes, we propose a novel SDF-based implicit mesh learning approach for 3D Gaussians that regularizes the underlying geometries and extracts highly detailed textured meshes. Our proposed method, GAvatar, enables the large-scale generation of diverse animatable avatars using only text prompts. GAvatar significantly surpasses existing methods in terms of both appearance and geometry quality, and achieves extremely fast rendering (100 fps) at 1K resolution.

Related papers

• SqueezeMe: Mobile-Ready Distillation of Gaussian Full-Body Avatars [19.249226899376943]
  We present SqueezeMe, a framework to convert high-fidelity 3D Gaussian full-body avatars into a lightweight representation. We achieve, for the first time, simultaneous animation and rendering of 3 Gaussian avatars in real-time (72 FPS) on a Meta Quest 3 VR headset.
  arXiv  Detail & Related papers  (2024-12-19T18:46:55Z)
• Textured Gaussians for Enhanced 3D Scene Appearance Modeling [58.134905268540436]
  3D Gaussian Splatting (3DGS) has emerged as a state-of-the-art 3D reconstruction and rendering technique. We propose a new generalized Gaussian appearance representation that augments each Gaussian with alpha(A), RGB, or RGBA texture maps. We demonstrate image quality improvements over existing methods while using a similar or lower number of Gaussians.
  arXiv  Detail & Related papers  (2024-11-27T18:59:59Z)
• Generalizable and Animatable Gaussian Head Avatar [50.34788590904843]
  We propose Generalizable and Animatable Gaussian head Avatar (GAGAvatar) for one-shot animatable head avatar reconstruction. We generate the parameters of 3D Gaussians from a single image in a single forward pass. Our method exhibits superior performance compared to previous methods in terms of reconstruction quality and expression accuracy.
  arXiv  Detail & Related papers  (2024-10-10T14:29:00Z)
• Gaussian Eigen Models for Human Heads [28.49783203616257]
  Current personalized neural head avatars face a trade-off: lightweight models lack detail and realism, while high-quality, animatable avatars require significant computational resources. We introduce Gaussian Eigen Models (GEM), which provide high-quality, lightweight, and easily controllable head avatars.
  arXiv  Detail & Related papers  (2024-07-05T14:30:24Z)
• $E^{3}$Gen: Efficient, Expressive and Editable Avatars Generation [71.72171053129655]
  This paper aims to introduce 3D Gaussian for efficient, expressive, and editable digital avatar generation. We propose a novel avatar generation method named $E3$Gen to effectively address these challenges. Our method achieves superior performance in avatar generation and enables expressive full-body pose control and editing.
  arXiv  Detail & Related papers  (2024-05-29T15:43:49Z)
• GaussianFormer: Scene as Gaussians for Vision-Based 3D Semantic Occupancy Prediction [70.65250036489128]
  3D semantic occupancy prediction aims to obtain 3D fine-grained geometry and semantics of the surrounding scene. We propose an object-centric representation to describe 3D scenes with sparse 3D semantic Gaussians. GaussianFormer achieves comparable performance with state-of-the-art methods with only 17.8% - 24.8% of their memory consumption.
  arXiv  Detail & Related papers  (2024-05-27T17:59:51Z)
• ASH: Animatable Gaussian Splats for Efficient and Photoreal Human Rendering [62.81677824868519]
  We propose an animatable Gaussian splatting approach for photorealistic rendering of dynamic humans in real-time. We parameterize the clothed human as animatable 3D Gaussians, which can be efficiently splatted into image space to generate the final rendering. We benchmark ASH with competing methods on pose-controllable avatars, demonstrating that our method outperforms existing real-time methods by a large margin and shows comparable or even better results than offline methods.
  arXiv  Detail & Related papers  (2023-12-10T17:07:37Z)
• GaussianHead: High-fidelity Head Avatars with Learnable Gaussian Derivation [35.39887092268696]
  This paper presents a framework to model the actional human head with anisotropic 3D Gaussians. In experiments, our method can produce high-fidelity renderings, outperforming state-of-the-art approaches in reconstruction, cross-identity reenactment, and novel view synthesis tasks.
  arXiv  Detail & Related papers  (2023-12-04T05:24:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.