MixedGaussianAvatar: Realistically and Geometrically Accurate Head Avatar via Mixed 2D-3D Gaussian Splatting

• URL: http://arxiv.org/abs/2412.04955v2
• Date: Wed, 11 Dec 2024 06:00:52 GMT
• Title: MixedGaussianAvatar: Realistically and Geometrically Accurate Head Avatar via Mixed 2D-3D Gaussian Splatting
• Authors: Peng Chen, Xiaobao Wei, Qingpo Wuwu, Xinyi Wang, Xingyu Xiao, Ming Lu,
• Abstract summary: Reconstructing high-fidelity 3D head avatars is crucial in various applications such as virtual reality.<n>Recent methods based on 3D Gaussian Splatting (3DGS) significantly improve the efficiency of training and rendering.<n>We propose a novel method named MixedGaussianAvatar for realistically and geometrically accurate head avatar reconstruction.
• Score: 38.16397253003339
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reconstructing high-fidelity 3D head avatars is crucial in various applications such as virtual reality. The pioneering methods reconstruct realistic head avatars with Neural Radiance Fields (NeRF), which have been limited by training and rendering speed. Recent methods based on 3D Gaussian Splatting (3DGS) significantly improve the efficiency of training and rendering. However, the surface inconsistency of 3DGS results in subpar geometric accuracy; later, 2DGS uses 2D surfels to enhance geometric accuracy at the expense of rendering fidelity. To leverage the benefits of both 2DGS and 3DGS, we propose a novel method named MixedGaussianAvatar for realistically and geometrically accurate head avatar reconstruction. Our main idea is to utilize 2D Gaussians to reconstruct the surface of the 3D head, ensuring geometric accuracy. We attach the 2D Gaussians to the triangular mesh of the FLAME model and connect additional 3D Gaussians to those 2D Gaussians where the rendering quality of 2DGS is inadequate, creating a mixed 2D-3D Gaussian representation. These 2D-3D Gaussians can then be animated using FLAME parameters. We further introduce a progressive training strategy that first trains the 2D Gaussians and then fine-tunes the mixed 2D-3D Gaussians. We demonstrate the superiority of MixedGaussianAvatar through comprehensive experiments. The code will be released at: https://github.com/ChenVoid/MGA/.

Related papers

• SHeaP: Self-Supervised Head Geometry Predictor Learned via 2D Gaussians [55.813327441814344]
  3D reconstruction of human heads from monocular images and videos underlies numerous visual applications. Previous methods have sought to learn from abundant 2D videos in a self-supervised manner. We propose SHeaP (Self-supervised Head Geometry Predictor Learned via 2D Gaussians)
  arXiv  Detail & Related papers  (2025-04-16T17:55:02Z)
• High-fidelity 3D Object Generation from Single Image with RGBN-Volume Gaussian Reconstruction Model [38.13429047918231]
  We propose a novel hybrid Voxel-Gaussian representation, where a 3D voxel representation contains explicit 3D geometric information. Our 3D voxel representation is obtained by a fusion module that aligns RGB features and surface normal features, both of which can be estimated from 2D images.
  arXiv  Detail & Related papers  (2025-04-02T08:58:34Z)
• GaussRender: Learning 3D Occupancy with Gaussian Rendering [86.89653628311565]
  GaussRender is a module that improves 3D occupancy learning by enforcing projective consistency. Our method penalizes 3D configurations that produce inconsistent 2D projections, thereby enforcing a more coherent 3D structure.
  arXiv  Detail & Related papers  (2025-02-07T16:07:51Z)
• MiraGe: Editable 2D Images using Gaussian Splatting [0.0]
  Implicit Neural Representations (INRs) approximate discrete data through continuous functions and are commonly used for encoding 2D images. We introduce a novel method, MiraGe, which uses mirror reflections to perceive 2D images in 3D space and employs flat-controlled Gaussians for precise 2D image editing. Our approach improves the rendering quality and allows realistic image modifications, including human-inspired perception of photos in the 3D world.
  arXiv  Detail & Related papers  (2024-10-02T13:10:57Z)
• 3D-GSW: 3D Gaussian Splatting for Robust Watermarking [5.52538716292462]
  We introduce a robust watermarking method for 3D-GS that secures ownership of both the model and its rendered images. Our proposed method remains robust against distortions in rendered images and model attacks while maintaining high rendering quality.
  arXiv  Detail & Related papers  (2024-09-20T05:16:06Z)
• PUP 3D-GS: Principled Uncertainty Pruning for 3D Gaussian Splatting [59.277480452459315]
  We propose a principled sensitivity pruning score that preserves visual fidelity and foreground details at significantly higher compression ratios.<n>We also propose a multi-round prune-refine pipeline that can be applied to any pretrained 3D-GS model without changing its training pipeline.
  arXiv  Detail & Related papers  (2024-06-14T17:53:55Z)
• LeanGaussian: Breaking Pixel or Point Cloud Correspondence in Modeling 3D Gaussians [12.408610403423559]
  We introduce LeanGaussian, a novel approach that treats each query in deformable Transformer as one 3D Gaussian ellipsoid.<n>We leverage deformable decoder to iteratively refine the Gaussians layer-by-layer with the image features as keys and values.<n>Our approach outperforms prior methods by approximately 6.1%, achieving a PSNR of 25.44 and 22.36, respectively.
  arXiv  Detail & Related papers  (2024-04-25T04:18:59Z)
• 2D Gaussian Splatting for Geometrically Accurate Radiance Fields [50.056790168812114]
  3D Gaussian Splatting (3DGS) has recently revolutionized radiance field reconstruction, achieving high quality novel view synthesis and fast rendering speed without baking. We present 2D Gaussian Splatting (2DGS), a novel approach to model and reconstruct geometrically accurate radiance fields from multi-view images. We demonstrate that our differentiable terms allows for noise-free and detailed geometry reconstruction while maintaining competitive appearance quality, fast training speed, and real-time rendering.
  arXiv  Detail & Related papers  (2024-03-26T17:21:24Z)
• ASH: Animatable Gaussian Splats for Efficient and Photoreal Human Rendering [62.81677824868519]
  We propose an animatable Gaussian splatting approach for photorealistic rendering of dynamic humans in real-time. We parameterize the clothed human as animatable 3D Gaussians, which can be efficiently splatted into image space to generate the final rendering. We benchmark ASH with competing methods on pose-controllable avatars, demonstrating that our method outperforms existing real-time methods by a large margin and shows comparable or even better results than offline methods.
  arXiv  Detail & Related papers  (2023-12-10T17:07:37Z)
• Gaussian3Diff: 3D Gaussian Diffusion for 3D Full Head Synthesis and Editing [53.05069432989608]
  We present a novel framework for generating 3D human heads with remarkable flexibility. Our method facilitates the creation of diverse and realistic 3D human heads with fine-grained editing over facial features and expressions.
  arXiv  Detail & Related papers  (2023-12-05T19:05:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.