Hybrid Explicit Representation for Ultra-Realistic Head Avatars

• URL: http://arxiv.org/abs/2403.11453v2
• Date: Wed, 19 Feb 2025 03:20:12 GMT
• Title: Hybrid Explicit Representation for Ultra-Realistic Head Avatars
• Authors: Hongrui Cai, Yuting Xiao, Xuan Wang, Jiafei Li, Yudong Guo, Yanbo Fan, Shenghua Gao, Juyong Zhang,
• Abstract summary: We introduce a novel approach to creating ultra-realistic head avatars and rendering them in real-time. UV-mapped 3D mesh is utilized to capture sharp and rich textures on smooth surfaces, while 3D Gaussian Splatting is employed to represent complex geometric structures. Experiments that our modeled results exceed those of state-of-the-art approaches.
• Score: 55.829497543262214
• License:
• Abstract: We introduce a novel approach to creating ultra-realistic head avatars and rendering them in real-time (>30fps at $2048 \times 1334$ resolution). First, we propose a hybrid explicit representation that combines the advantages of two primitive-based efficient rendering techniques. UV-mapped 3D mesh is utilized to capture sharp and rich textures on smooth surfaces, while 3D Gaussian Splatting is employed to represent complex geometric structures. In the pipeline of modeling an avatar, after tracking parametric models based on captured multi-view RGB videos, our goal is to simultaneously optimize the texture and opacity map of mesh, as well as a set of 3D Gaussian splats localized and rigged onto the mesh facets. Specifically, we perform $\alpha$-blending on the color and opacity values based on the merged and re-ordered z-buffer from the rasterization results of mesh and 3DGS. This process involves the mesh and 3DGS adaptively fitting the captured visual information to outline a high-fidelity digital avatar. To avoid artifacts caused by Gaussian splats crossing the mesh facets, we design a stable hybrid depth sorting strategy. Experiments illustrate that our modeled results exceed those of state-of-the-art approaches.

Related papers

• 3D Gaussian Splatting with Normal Information for Mesh Extraction and Improved Rendering [8.59572577251833]
  We propose a novel regularization method using the gradients of a signed distance function estimated from the Gaussians. We demonstrate the effectiveness of our approach on datasets such as Mip-NeRF360, Tanks and Temples, and Deep-Blending.
  arXiv  Detail & Related papers  (2025-01-14T18:40:33Z)
• 3D Convex Splatting: Radiance Field Rendering with 3D Smooth Convexes [87.01284850604495]
  We introduce 3D Convexting (3DCS), which leverages 3D smooth convexes as primitives for modeling geometrically-meaningful radiance fields from multiview images. 3DCS achieves superior performance over 3DGS on benchmarks such as MipNeizer, Tanks and Temples, and Deep Blending. Our results highlight the potential of 3D Convexting to become the new standard for high-quality scene reconstruction.
  arXiv  Detail & Related papers  (2024-11-22T14:31:39Z)
• GUS-IR: Gaussian Splatting with Unified Shading for Inverse Rendering [83.69136534797686]
  We present GUS-IR, a novel framework designed to address the inverse rendering problem for complicated scenes featuring rough and glossy surfaces. This paper starts by analyzing and comparing two prominent shading techniques popularly used for inverse rendering, forward shading and deferred shading. We propose a unified shading solution that combines the advantages of both techniques for better decomposition.
  arXiv  Detail & Related papers  (2024-11-12T01:51:05Z)
• MeshGS: Adaptive Mesh-Aligned Gaussian Splatting for High-Quality Rendering [61.64903786502728]
  We propose a novel approach that integrates mesh representation with 3D Gaussian splats to perform high-quality rendering of reconstructed real-world scenes. We consider the distance between each Gaussian splat and the mesh surface to distinguish between tightly-bound and loosely-bound splats. Our method surpasses recent mesh-based neural rendering techniques by achieving a 2dB higher PSNR, and outperforms mesh-based Gaussian splatting methods by 1.3 dB PSNR.
  arXiv  Detail & Related papers  (2024-10-11T16:07:59Z)
• DreamMesh4D: Video-to-4D Generation with Sparse-Controlled Gaussian-Mesh Hybrid Representation [10.250715657201363]
  We introduce DreamMesh4D, a novel framework combining mesh representation with geometric skinning technique to generate high-quality 4D object from a monocular video. Our method is compatible with modern graphic pipelines, showcasing its potential in the 3D gaming and film industry.
  arXiv  Detail & Related papers  (2024-10-09T10:41:08Z)
• SpecGaussian with Latent Features: A High-quality Modeling of the View-dependent Appearance for 3D Gaussian Splatting [11.978842116007563]
  Lantent-SpecGS is an approach that utilizes a universal latent neural descriptor within each 3D Gaussian. Two parallel CNNs are designed to decoder the splatting feature maps into diffuse color and specular color separately. A mask that depends on the viewpoint is learned to merge these two colors, resulting in the final rendered image.
  arXiv  Detail & Related papers  (2024-08-23T15:25:08Z)
• GSD: View-Guided Gaussian Splatting Diffusion for 3D Reconstruction [52.04103235260539]
  We present a diffusion model approach based on Gaussian Splatting representation for 3D object reconstruction from a single view. The model learns to generate 3D objects represented by sets of GS ellipsoids. The final reconstructed objects explicitly come with high-quality 3D structure and texture, and can be efficiently rendered in arbitrary views.
  arXiv  Detail & Related papers  (2024-07-05T03:43:08Z)
• SplattingAvatar: Realistic Real-Time Human Avatars with Mesh-Embedded Gaussian Splatting [26.849406891462557]
  We present SplattingAvatar, a hybrid 3D representation of human avatars with Gaussian Splatting embedded on a triangle mesh. SplattingAvatar renders over 300 FPS on a modern GPU and 30 FPS on a mobile device.
  arXiv  Detail & Related papers  (2024-03-08T06:28:09Z)
• GETAvatar: Generative Textured Meshes for Animatable Human Avatars [69.56959932421057]
  We study the problem of 3D-aware full-body human generation, aiming at creating animatable human avatars with high-quality geometries and textures. We propose GETAvatar, a Generative model that directly generates Explicit Textured 3D rendering for animatable human Avatar.
  arXiv  Detail & Related papers  (2023-10-04T10:30:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.