Single-Camera 3D Head Fitting for Mixed Reality Clinical Applications

• URL: http://arxiv.org/abs/2109.02740v1
• Date: Mon, 6 Sep 2021 21:03:52 GMT
• Title: Single-Camera 3D Head Fitting for Mixed Reality Clinical Applications
• Authors: Tejas Mane, Aylar Bayramova, Kostas Daniilidis, Philippos Mordohai, Elena Bernardis
• Abstract summary: Our goal is to reconstruct the head model of each person to enable future mixed reality applications. We recover a dense 3D reconstruction and camera information via structure-from-motion and multi-view stereo. These are then used in a new two-stage fitting process to recover the 3D head shape.
• Score: 41.63137498124499
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We address the problem of estimating the shape of a person's head, defined as the geometry of the complete head surface, from a video taken with a single moving camera, and determining the alignment of the fitted 3D head for all video frames, irrespective of the person's pose. 3D head reconstructions commonly tend to focus on perfecting the face reconstruction, leaving the scalp to a statistical approximation. Our goal is to reconstruct the head model of each person to enable future mixed reality applications. To do this, we recover a dense 3D reconstruction and camera information via structure-from-motion and multi-view stereo. These are then used in a new two-stage fitting process to recover the 3D head shape by iteratively fitting a 3D morphable model of the head with the dense reconstruction in canonical space and fitting it to each person's head, using both traditional facial landmarks and scalp features extracted from the head's segmentation mask. Our approach recovers consistent geometry for varying head shapes, from videos taken by different people, with different smartphones, and in a variety of environments from living rooms to outdoor spaces.

Related papers

• Head Pose Estimation and 3D Neural Surface Reconstruction via Monocular Camera in situ for Navigation and Safe Insertion into Natural Openings [4.592222359553848]
  We have chosen 3D Slicer as our base platform and monocular cameras are used as sensors. We used the neural radiance fields (NeRF) algorithm to complete the 3D model reconstruction of the human head. The individual's head pose, obtained through single-camera vision, is transmitted in real-time to the scene created within 3D Slicer.
  arXiv  Detail & Related papers  (2024-06-18T20:42:09Z)
• HeadRecon: High-Fidelity 3D Head Reconstruction from Monocular Video [37.53752896927615]
  We study the reconstruction of high-fidelity 3D head models from arbitrary monocular videos. We propose a prior-guided dynamic implicit neural network to tackle these problems.
  arXiv  Detail & Related papers  (2023-12-14T12:38:56Z)
• Decaf: Monocular Deformation Capture for Face and Hand Interactions [77.75726740605748]
  This paper introduces the first method that allows tracking human hands interacting with human faces in 3D from single monocular RGB videos. We model hands as articulated objects inducing non-rigid face deformations during an active interaction. Our method relies on a new hand-face motion and interaction capture dataset with realistic face deformations acquired with a markerless multi-view camera system.
  arXiv  Detail & Related papers  (2023-09-28T17:59:51Z)
• Generalizable One-shot Neural Head Avatar [90.50492165284724]
  We present a method that reconstructs and animates a 3D head avatar from a single-view portrait image. We propose a framework that not only generalizes to unseen identities based on a single-view image, but also captures characteristic details within and beyond the face area.
  arXiv  Detail & Related papers  (2023-06-14T22:33:09Z)
• Head3D: Complete 3D Head Generation via Tri-plane Feature Distillation [56.267877301135634]
  Current full head generation methods require a large number of 3D scans or multi-view images to train the model. We propose Head3D, a method to generate full 3D heads with limited multi-view images. Our model achieves cost-efficient and diverse complete head generation with photo-realistic renderings and high-quality geometry representations.
  arXiv  Detail & Related papers  (2023-03-28T11:12:26Z)
• Learning to regulate 3D head shape by removing occluding hair from in-the-wild images [0.0]
  We present a novel approach for modeling the upper head by removing occluding hair and reconstructing the skin. Our unsupervised 3DMM model achieves state-of-the-art results on popular benchmarks.
  arXiv  Detail & Related papers  (2022-08-25T13:18:26Z)
• Prior-Guided Multi-View 3D Head Reconstruction [28.126115947538572]
  Previous multi-view stereo methods suffer from low-frequency structures such as unclear head structures and inaccurate reconstruction in hair regions. To tackle this problem, we propose a prior-guided implicit neural rendering network. The utilization of these priors can improve the reconstruction accuracy and robustness, leading to a high-quality integrated 3D head model.
  arXiv  Detail & Related papers  (2021-07-09T07:43:56Z)
• HeadGAN: One-shot Neural Head Synthesis and Editing [70.30831163311296]
  HeadGAN is a system that synthesises on 3D face representations and adapted to the facial geometry of any reference image. The 3D face representation enables HeadGAN to be further used as an efficient method for compression and reconstruction and a tool for expression and pose editing.
  arXiv  Detail & Related papers  (2020-12-15T12:51:32Z)
• Deep 3D Portrait from a Single Image [54.634207317528364]
  We present a learning-based approach for recovering the 3D geometry of human head from a single portrait image. A two-step geometry learning scheme is proposed to learn 3D head reconstruction from in-the-wild face images. We evaluate the accuracy of our method both in 3D and with pose manipulation tasks on 2D images.
  arXiv  Detail & Related papers  (2020-04-24T08:55:37Z)
• Real-time Simultaneous 3D Head Modeling and Facial Motion Capture with an RGB-D camera [2.3260877354419254]
  We propose a method to build in real-time animated 3D head models using a consumer-grade RGB-D camera. Anyone's head can be instantly reconstructed and his facial motion captured without requiring any training or pre-scanning.
  arXiv  Detail & Related papers  (2020-04-22T13:22:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.