Neural Point-based Volumetric Avatar: Surface-guided Neural Points for Efficient and Photorealistic Volumetric Head Avatar

• URL: http://arxiv.org/abs/2307.05000v2
• Date: Fri, 13 Oct 2023 11:07:09 GMT
• Title: Neural Point-based Volumetric Avatar: Surface-guided Neural Points for Efficient and Photorealistic Volumetric Head Avatar
• Authors: Cong Wang, Di Kang, Yan-Pei Cao, Linchao Bao, Ying Shan, Song-Hai Zhang
• Abstract summary: We propose fullname (name), a method that adopts the neural point representation and the neural volume rendering process. Specifically, the neural points are strategically constrained around the surface of the target expression via a high-resolution UV displacement map. By design, our name is better equipped to handle topologically changing regions and thin structures while also ensuring accurate expression control when animating avatars.
• Score: 62.87222308616711
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Rendering photorealistic and dynamically moving human heads is crucial for ensuring a pleasant and immersive experience in AR/VR and video conferencing applications. However, existing methods often struggle to model challenging facial regions (e.g., mouth interior, eyes, hair/beard), resulting in unrealistic and blurry results. In this paper, we propose {\fullname} ({\name}), a method that adopts the neural point representation as well as the neural volume rendering process and discards the predefined connectivity and hard correspondence imposed by mesh-based approaches. Specifically, the neural points are strategically constrained around the surface of the target expression via a high-resolution UV displacement map, achieving increased modeling capacity and more accurate control. We introduce three technical innovations to improve the rendering and training efficiency: a patch-wise depth-guided (shading point) sampling strategy, a lightweight radiance decoding process, and a Grid-Error-Patch (GEP) ray sampling strategy during training. By design, our {\name} is better equipped to handle topologically changing regions and thin structures while also ensuring accurate expression control when animating avatars. Experiments conducted on three subjects from the Multiface dataset demonstrate the effectiveness of our designs, outperforming previous state-of-the-art methods, especially in handling challenging facial regions.

Related papers

• Efficient 3D Implicit Head Avatar with Mesh-anchored Hash Table Blendshapes [40.35875246929843]
  3D head avatars built with neural implicit representations have achieved unprecedented levels of photorealism. We propose a novel fast 3D neural implicit head avatar model that achieves real-time rendering while maintaining fine-grained controllability and high rendering quality.
  arXiv  Detail & Related papers  (2024-04-02T00:55:50Z)
• Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering [71.44349029439944]
  Recent 3D Gaussian Splatting method has achieved the state-of-the-art rendering quality and speed. We introduce Scaffold-GS, which uses anchor points to distribute local 3D Gaussians. We show that our method effectively reduces redundant Gaussians while delivering high-quality rendering.
  arXiv  Detail & Related papers  (2023-11-30T17:58:57Z)
• Leveraging Neural Radiance Fields for Uncertainty-Aware Visual Localization [56.95046107046027]
  We propose to leverage Neural Radiance Fields (NeRF) to generate training samples for scene coordinate regression. Despite NeRF's efficiency in rendering, many of the rendered data are polluted by artifacts or only contain minimal information gain.
  arXiv  Detail & Related papers  (2023-10-10T20:11:13Z)
• High-Fidelity 3D Head Avatars Reconstruction through Spatially-Varying Expression Conditioned Neural Radiance Field [44.37605022793316]
  We introduce a novel Spatially-Varying Expression (SVE) conditioning. The proposed SVE-conditioned neural radiance field can deal with intricate facial expressions and achieve realistic rendering and geometry details of high-fidelity 3D head avatars. Our method outperforms other state-of-the-art (SOTA) methods in rendering and geometry quality on mobile phone-collected and public datasets.
  arXiv  Detail & Related papers  (2023-10-10T03:13:33Z)
• Learning Personalized High Quality Volumetric Head Avatars from Monocular RGB Videos [47.94545609011594]
  We propose a method to learn a high-quality implicit 3D head avatar from a monocular RGB video captured in the wild. Our hybrid pipeline combines the geometry prior and dynamic tracking of a 3DMM with a neural radiance field to achieve fine-grained control and photorealism.
  arXiv  Detail & Related papers  (2023-04-04T01:10:04Z)
• Neural 3D Reconstruction in the Wild [86.6264706256377]
  We introduce a new method that enables efficient and accurate surface reconstruction from Internet photo collections. We present a new benchmark and protocol for evaluating reconstruction performance on such in-the-wild scenes.
  arXiv  Detail & Related papers  (2022-05-25T17:59:53Z)
• Inverting Generative Adversarial Renderer for Face Reconstruction [58.45125455811038]
  In this work, we introduce a novel Generative Adversa Renderer (GAR) GAR learns to model the complicated real-world image, instead of relying on the graphics rules, it is capable of producing realistic images. Our method achieves state-of-the-art performances on multiple face reconstruction.
  arXiv  Detail & Related papers  (2021-05-06T04:16:06Z)
• Neural Lumigraph Rendering [33.676795978166375]
  State-of-the-art (SOTA) neural volume rendering approaches are slow to train and require minutes of inference (i.e., rendering) time for high image resolutions. We adopt high-capacity neural scene representations with periodic activations for jointly optimizing an implicit surface and a radiance field of a scene supervised exclusively with posed 2D images. Our neural rendering pipeline accelerates SOTA neural volume rendering by about two orders of magnitude and our implicit surface representation is unique in allowing us to export a mesh with view-dependent texture information.
  arXiv  Detail & Related papers  (2021-03-22T03:46:05Z)
• PVA: Pixel-aligned Volumetric Avatars [34.929560973779466]
  We devise a novel approach for predicting volumetric avatars of the human head given just a small number of inputs. Our approach is trained in an end-to-end manner solely based on a photometric re-rendering loss without requiring explicit 3D supervision.
  arXiv  Detail & Related papers  (2021-01-07T18:58:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.