NPGA: Neural Parametric Gaussian Avatars

• URL: http://arxiv.org/abs/2405.19331v2
• Date: Fri, 13 Sep 2024 17:41:21 GMT
• Title: NPGA: Neural Parametric Gaussian Avatars
• Authors: Simon Giebenhain, Tobias Kirschstein, Martin Rünz, Lourdes Agapito, Matthias Nießner,
• Abstract summary: We propose a data-driven approach to create high-fidelity controllable avatars from multi-view video recordings. We build our method around 3D Gaussian splatting for its highly efficient rendering and to inherit the topological flexibility of point clouds. We evaluate our method on the public NeRSemble dataset, demonstrating that NPGA significantly outperforms the previous state-of-the-art avatars on the self-reenactment task by 2.6 PSNR.
• Score: 46.52887358194364
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The creation of high-fidelity, digital versions of human heads is an important stepping stone in the process of further integrating virtual components into our everyday lives. Constructing such avatars is a challenging research problem, due to a high demand for photo-realism and real-time rendering performance. In this work, we propose Neural Parametric Gaussian Avatars (NPGA), a data-driven approach to create high-fidelity, controllable avatars from multi-view video recordings. We build our method around 3D Gaussian splatting for its highly efficient rendering and to inherit the topological flexibility of point clouds. In contrast to previous work, we condition our avatars' dynamics on the rich expression space of neural parametric head models (NPHM), instead of mesh-based 3DMMs. To this end, we distill the backward deformation field of our underlying NPHM into forward deformations which are compatible with rasterization-based rendering. All remaining fine-scale, expression-dependent details are learned from the multi-view videos. For increased representational capacity of our avatars, we propose per-Gaussian latent features that condition each primitives dynamic behavior. To regularize this increased dynamic expressivity, we propose Laplacian terms on the latent features and predicted dynamics. We evaluate our method on the public NeRSemble dataset, demonstrating that NPGA significantly outperforms the previous state-of-the-art avatars on the self-reenactment task by 2.6 PSNR. Furthermore, we demonstrate accurate animation capabilities from real-world monocular videos.

Related papers

• Bundle Adjusted Gaussian Avatars Deblurring [31.718130377229482]
  We propose a 3D-aware, physics-oriented model of blur formation attributable to human movement and a 3D human motion model to clarify ambiguities found in motion-induced blurry images. We have established benchmarks for this task through a synthetic dataset derived from existing multi-view captures, alongside a real-captured dataset acquired through a 360-degree synchronous hybrid-exposure camera system.
  arXiv  Detail & Related papers  (2024-11-24T10:03:24Z)
• Generalizable and Animatable Gaussian Head Avatar [50.34788590904843]
  We propose Generalizable and Animatable Gaussian head Avatar (GAGAvatar) for one-shot animatable head avatar reconstruction. We generate the parameters of 3D Gaussians from a single image in a single forward pass. Our method exhibits superior performance compared to previous methods in terms of reconstruction quality and expression accuracy.
  arXiv  Detail & Related papers  (2024-10-10T14:29:00Z)
• GPHM: Gaussian Parametric Head Model for Monocular Head Avatar Reconstruction [47.113910048252805]
  High-fidelity 3D human head avatars are crucial for applications in VR/AR, digital human, and film production. Recent advances have leveraged morphable face models to generate animated head avatars, representing varying identities and expressions. We introduce 3D Gaussian Parametric Head Model, which employs 3D Gaussians to accurately represent the complexities of the human head.
  arXiv  Detail & Related papers  (2024-07-21T06:03:11Z)
• Expressive Gaussian Human Avatars from Monocular RGB Video [69.56388194249942]
  We introduce EVA, a drivable human model that meticulously sculpts fine details based on 3D Gaussians and SMPL-X. We highlight the critical importance of aligning the SMPL-X model with RGB frames for effective avatar learning. We propose a context-aware adaptive density control strategy, which is adaptively adjusting the gradient thresholds.
  arXiv  Detail & Related papers  (2024-07-03T15:36:27Z)
• Deformable 3D Gaussian Splatting for Animatable Human Avatars [50.61374254699761]
  We propose a fully explicit approach to construct a digital avatar from as little as a single monocular sequence. ParDy-Human constitutes an explicit model for realistic dynamic human avatars which requires significantly fewer training views and images. Our avatars learning is free of additional annotations such as Splat masks and can be trained with variable backgrounds while inferring full-resolution images efficiently even on consumer hardware.
  arXiv  Detail & Related papers  (2023-12-22T20:56:46Z)
• GaussianAvatar: Towards Realistic Human Avatar Modeling from a Single Video via Animatable 3D Gaussians [51.46168990249278]
  We present an efficient approach to creating realistic human avatars with dynamic 3D appearances from a single video. GustafAvatar is validated on both the public dataset and our collected dataset.
  arXiv  Detail & Related papers  (2023-12-04T18:55:45Z)
• Learning Personalized High Quality Volumetric Head Avatars from Monocular RGB Videos [47.94545609011594]
  We propose a method to learn a high-quality implicit 3D head avatar from a monocular RGB video captured in the wild. Our hybrid pipeline combines the geometry prior and dynamic tracking of a 3DMM with a neural radiance field to achieve fine-grained control and photorealism.
  arXiv  Detail & Related papers  (2023-04-04T01:10:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.