Gaussian Deja-vu: Creating Controllable 3D Gaussian Head-Avatars with Enhanced Generalization and Personalization Abilities

• URL: http://arxiv.org/abs/2409.16147v3
• Date: Wed, 6 Nov 2024 18:08:23 GMT
• Title: Gaussian Deja-vu: Creating Controllable 3D Gaussian Head-Avatars with Enhanced Generalization and Personalization Abilities
• Authors: Peizhi Yan, Rabab Ward, Qiang Tang, Shan Du,
• Abstract summary: We introduce the "Gaussian Deja-vu" framework, which first obtains a generalized model of the head avatar and then personalizes the result. For personalizing, we propose learnable expression-aware rectification blendmaps, ensuring rapid convergence without the reliance on neural networks. It outperforms state-of-the-art 3D Gaussian head avatars in terms of photorealistic quality as well as reduces training time consumption to at least a quarter of the existing methods.
• Score: 10.816370283498287
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advancements in 3D Gaussian Splatting (3DGS) have unlocked significant potential for modeling 3D head avatars, providing greater flexibility than mesh-based methods and more efficient rendering compared to NeRF-based approaches. Despite these advancements, the creation of controllable 3DGS-based head avatars remains time-intensive, often requiring tens of minutes to hours. To expedite this process, we here introduce the "Gaussian Deja-vu" framework, which first obtains a generalized model of the head avatar and then personalizes the result. The generalized model is trained on large 2D (synthetic and real) image datasets. This model provides a well-initialized 3D Gaussian head that is further refined using a monocular video to achieve the personalized head avatar. For personalizing, we propose learnable expression-aware rectification blendmaps to correct the initial 3D Gaussians, ensuring rapid convergence without the reliance on neural networks. Experiments demonstrate that the proposed method meets its objectives. It outperforms state-of-the-art 3D Gaussian head avatars in terms of photorealistic quality as well as reduces training time consumption to at least a quarter of the existing methods, producing the avatar in minutes.

Related papers

• Generalizable and Animatable Gaussian Head Avatar [50.34788590904843]
  We propose Generalizable and Animatable Gaussian head Avatar (GAGAvatar) for one-shot animatable head avatar reconstruction. We generate the parameters of 3D Gaussians from a single image in a single forward pass. Our method exhibits superior performance compared to previous methods in terms of reconstruction quality and expression accuracy.
  arXiv  Detail & Related papers  (2024-10-10T14:29:00Z)
• GPHM: Gaussian Parametric Head Model for Monocular Head Avatar Reconstruction [47.113910048252805]
  High-fidelity 3D human head avatars are crucial for applications in VR/AR, digital human, and film production. Recent advances have leveraged morphable face models to generate animated head avatars, representing varying identities and expressions. We introduce 3D Gaussian Parametric Head Model, which employs 3D Gaussians to accurately represent the complexities of the human head.
  arXiv  Detail & Related papers  (2024-07-21T06:03:11Z)
• Gaussian Eigen Models for Human Heads [28.49783203616257]
  We present personalized Gaussian Eigen Models (GEMs) for human heads, a novel method that compresses dynamic 3D Gaussians into low-dimensional linear spaces. Our approach is inspired by the seminal work of Blanz and Vetter, where a mesh-based 3D morphable model (3DMM) is constructed from registered meshes. We show and compare self-reenactment and cross-person reenactment to state-of-the-art 3D avatar methods, demonstrating higher quality and better control.
  arXiv  Detail & Related papers  (2024-07-05T14:30:24Z)
• GVA: Reconstructing Vivid 3D Gaussian Avatars from Monocular Videos [56.40776739573832]
  We present a novel method that facilitates the creation of vivid 3D Gaussian avatars from monocular video inputs (GVA) Our innovation lies in addressing the intricate challenges of delivering high-fidelity human body reconstructions. We introduce a pose refinement technique to improve hand and foot pose accuracy by aligning normal maps and silhouettes.
  arXiv  Detail & Related papers  (2024-02-26T14:40:15Z)
• GAvatar: Animatable 3D Gaussian Avatars with Implicit Mesh Learning [60.33970027554299]
  Gaussian splatting has emerged as a powerful 3D representation that harnesses the advantages of both explicit (mesh) and implicit (NeRF) 3D representations. In this paper, we seek to leverage Gaussian splatting to generate realistic animatable avatars from textual descriptions. Our proposed method, GAvatar, enables the large-scale generation of diverse animatable avatars using only text prompts.
  arXiv  Detail & Related papers  (2023-12-18T18:59:12Z)
• ASH: Animatable Gaussian Splats for Efficient and Photoreal Human Rendering [62.81677824868519]
  We propose an animatable Gaussian splatting approach for photorealistic rendering of dynamic humans in real-time. We parameterize the clothed human as animatable 3D Gaussians, which can be efficiently splatted into image space to generate the final rendering. We benchmark ASH with competing methods on pose-controllable avatars, demonstrating that our method outperforms existing real-time methods by a large margin and shows comparable or even better results than offline methods.
  arXiv  Detail & Related papers  (2023-12-10T17:07:37Z)
• Gaussian3Diff: 3D Gaussian Diffusion for 3D Full Head Synthesis and Editing [53.05069432989608]
  We present a novel framework for generating 3D human heads with remarkable flexibility. Our method facilitates the creation of diverse and realistic 3D human heads with fine-grained editing over facial features and expressions.
  arXiv  Detail & Related papers  (2023-12-05T19:05:58Z)
• HeadGaS: Real-Time Animatable Head Avatars via 3D Gaussian Splatting [11.849852156716171]
  HeadGaS is a model that uses 3D Gaussian Splats (3DGS) for 3D head reconstruction and animation. We demonstrate that HeadGaS delivers state-of-the-art results in real-time inference frame rates, surpassing baselines by up to 2dB.
  arXiv  Detail & Related papers  (2023-12-05T17:19:22Z)
• Gaussian Head Avatar: Ultra High-fidelity Head Avatar via Dynamic Gaussians [41.86540576028268]
  We propose controllable 3D Gaussian Head Avatars for lightweight sparse-view setups. We show our approach outperforms other state-of-the-art sparse-view methods, achieving ultra high-fidelity rendering quality at 2K resolution even under exaggerated expressions.
  arXiv  Detail & Related papers  (2023-12-05T11:01:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.