Multi-NeuS: 3D Head Portraits from Single Image with Neural Implicit Functions

• URL: http://arxiv.org/abs/2209.04436v2
• Date: Fri, 8 Sep 2023 21:33:11 GMT
• Title: Multi-NeuS: 3D Head Portraits from Single Image with Neural Implicit Functions
• Authors: Egor Burkov, Ruslan Rakhimov, Aleksandr Safin, Evgeny Burnaev, Victor Lempitsky
• Abstract summary: We present an approach for the reconstruction of 3D human heads from one or few views. The underlying neural architecture is to learn the objects and to generalize the model. Our model can fit novel heads on just a hundred videos or one-shot 3D scans.
• Score: 70.04394678730968
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present an approach for the reconstruction of textured 3D meshes of human heads from one or few views. Since such few-shot reconstruction is underconstrained, it requires prior knowledge which is hard to impose on traditional 3D reconstruction algorithms. In this work, we rely on the recently introduced 3D representation $\unicode{x2013}$ neural implicit functions $\unicode{x2013}$ which, being based on neural networks, allows to naturally learn priors about human heads from data, and is directly convertible to textured mesh. Namely, we extend NeuS, a state-of-the-art neural implicit function formulation, to represent multiple objects of a class (human heads in our case) simultaneously. The underlying neural net architecture is designed to learn the commonalities among these objects and to generalize to unseen ones. Our model is trained on just a hundred smartphone videos and does not require any scanned 3D data. Afterwards, the model can fit novel heads in the few-shot or one-shot modes with good results.

Related papers

• Primitive-based 3D Human-Object Interaction Modelling and Programming [59.47308081630886]
  We propose a novel 3D geometric primitive-based language to encode both humans and objects. We build a new benchmark on 3D HAOI consisting of primitives together with their images. We believe this primitive-based 3D HAOI representation would pave the way for 3D HAOI studies.
  arXiv  Detail & Related papers  (2023-12-17T13:16:49Z)
• PonderV2: Pave the Way for 3D Foundation Model with A Universal Pre-training Paradigm [114.47216525866435]
  We introduce a novel universal 3D pre-training framework designed to facilitate the acquisition of efficient 3D representation. For the first time, PonderV2 achieves state-of-the-art performance on 11 indoor and outdoor benchmarks, implying its effectiveness.
  arXiv  Detail & Related papers  (2023-10-12T17:59:57Z)
• Approaching human 3D shape perception with neurally mappable models [15.090436065092716]
  Humans effortlessly infer the 3D shape of objects. None of current computational models capture the human ability to match object shape across viewpoints. This work provides a foundation for understanding human shape inferences within neurally mappable computational architectures.
  arXiv  Detail & Related papers  (2023-08-22T09:29:05Z)
• HQ3DAvatar: High Quality Controllable 3D Head Avatar [65.70885416855782]
  This paper presents a novel approach to building highly photorealistic digital head avatars. Our method learns a canonical space via an implicit function parameterized by a neural network. At test time, our method is driven by a monocular RGB video.
  arXiv  Detail & Related papers  (2023-03-25T13:56:33Z)
• PanoHead: Geometry-Aware 3D Full-Head Synthesis in 360$^{\circ}$ [17.355141949293852]
  Existing 3D generative adversarial networks (GANs) for 3D human head synthesis are either limited to near-frontal views or hard to preserve 3D consistency in large view angles. We propose PanoHead, the first 3D-aware generative model that enables high-quality view-consistent image synthesis of full heads in $360circ$ with diverse appearance and detailed geometry.
  arXiv  Detail & Related papers  (2023-03-23T06:54:34Z)
• Reconstructing Personalized Semantic Facial NeRF Models From Monocular Video [27.36067360218281]
  We present a novel semantic model for human head defined with neural radiance field. The 3D-consistent head model consist of a set of disentangled and interpretable bases, and can be driven by low-dimensional expression coefficients. With a short monocular RGB video as input, our method can construct the subject's semantic facial NeRF model with only ten to twenty minutes.
  arXiv  Detail & Related papers  (2022-10-12T11:56:52Z)
• Neural Novel Actor: Learning a Generalized Animatable Neural Representation for Human Actors [98.24047528960406]
  We propose a new method for learning a generalized animatable neural representation from a sparse set of multi-view imagery of multiple persons. The learned representation can be used to synthesize novel view images of an arbitrary person from a sparse set of cameras, and further animate them with the user's pose control.
  arXiv  Detail & Related papers  (2022-08-25T07:36:46Z)
• Free-HeadGAN: Neural Talking Head Synthesis with Explicit Gaze Control [54.079327030892244]
  Free-HeadGAN is a person-generic neural talking head synthesis system. We show that modeling faces with sparse 3D facial landmarks are sufficient for achieving state-of-the-art generative performance.
  arXiv  Detail & Related papers  (2022-08-03T16:46:08Z)
• ViewFormer: NeRF-free Neural Rendering from Few Images Using Transformers [34.4824364161812]
  Novel view synthesis is a problem where we are given only a few context views sparsely covering a scene or an object. The goal is to predict novel viewpoints in the scene, which requires learning priors. We propose a 2D-only method that maps multiple context views and a query pose to a new image in a single pass of a neural network.
  arXiv  Detail & Related papers  (2022-03-18T21:08:23Z)
• NeMo: Neural Mesh Models of Contrastive Features for Robust 3D Pose Estimation [11.271053492520535]
  3D pose estimation is a challenging but important task in computer vision. We show that standard deep learning approaches to 3D pose estimation are not robust when objects are partially occluded or viewed from a previously unseen pose. We propose to integrate deep neural networks with 3D generative representations of objects into a unified neural architecture that we term NeMo.
  arXiv  Detail & Related papers  (2021-01-29T03:23:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.