Photorealistic Monocular 3D Reconstruction of Humans Wearing Clothing

• URL: http://arxiv.org/abs/2204.08906v1
• Date: Tue, 19 Apr 2022 14:06:16 GMT
• Title: Photorealistic Monocular 3D Reconstruction of Humans Wearing Clothing
• Authors: Thiemo Alldieck, Mihai Zanfir, Cristian Sminchisescu
• Abstract summary: We present PHORHUM, a novel, end-to-end trainable, deep neural network methodology for photorealistic 3D human reconstruction given just a monocular RGB image. Our pixel-aligned method estimates detailed 3D geometry and, for the first time, the unshaded surface color together with the scene illumination.
• Score: 41.34640834483265
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present PHORHUM, a novel, end-to-end trainable, deep neural network methodology for photorealistic 3D human reconstruction given just a monocular RGB image. Our pixel-aligned method estimates detailed 3D geometry and, for the first time, the unshaded surface color together with the scene illumination. Observing that 3D supervision alone is not sufficient for high fidelity color reconstruction, we introduce patch-based rendering losses that enable reliable color reconstruction on visible parts of the human, and detailed and plausible color estimation for the non-visible parts. Moreover, our method specifically addresses methodological and practical limitations of prior work in terms of representing geometry, albedo, and illumination effects, in an end-to-end model where factors can be effectively disentangled. In extensive experiments, we demonstrate the versatility and robustness of our approach. Our state-of-the-art results validate the method qualitatively and for different metrics, for both geometric and color reconstruction.

Related papers

• EEG-Driven 3D Object Reconstruction with Color Consistency and Diffusion Prior [1.7205106391379026]
  We propose a method for reconstructing 3D objects with color consistency based on EEG signals. We demonstrate that our method can reconstruct 3D objects with color consistency using EEG.
  arXiv  Detail & Related papers  (2024-10-28T12:59:24Z)
• ANIM: Accurate Neural Implicit Model for Human Reconstruction from a single RGB-D image [40.03212588672639]
  ANIM is a novel method that reconstructs arbitrary 3D human shapes from single-view RGB-D images with an unprecedented level of accuracy. Our model learns geometric details from both pixel-aligned and voxel-aligned features to leverage depth information. Experiments demonstrate that ANIM outperforms state-of-the-art works that use RGB, surface normals, point cloud or RGB-D data as input.
  arXiv  Detail & Related papers  (2024-03-15T14:45:38Z)
• Multi-View Neural Surface Reconstruction with Structured Light [7.709526244898887]
  Three-dimensional (3D) object reconstruction based on differentiable rendering (DR) is an active research topic in computer vision. We introduce active sensing with structured light (SL) into multi-view 3D object reconstruction based on DR to learn the unknown geometry and appearance of arbitrary scenes and camera poses. Our method realizes high reconstruction accuracy in the textureless region and reduces efforts for camera pose calibration.
  arXiv  Detail & Related papers  (2022-11-22T03:10:46Z)
• ReFu: Refine and Fuse the Unobserved View for Detail-Preserving Single-Image 3D Human Reconstruction [31.782985891629448]
  Single-image 3D human reconstruction aims to reconstruct the 3D textured surface of the human body given a single image. We propose ReFu, a coarse-to-fine approach that refines the projected backside view image and fuses the refined image to predict the final human body.
  arXiv  Detail & Related papers  (2022-11-09T09:14:11Z)
• Neural 3D Reconstruction in the Wild [86.6264706256377]
  We introduce a new method that enables efficient and accurate surface reconstruction from Internet photo collections. We present a new benchmark and protocol for evaluating reconstruction performance on such in-the-wild scenes.
  arXiv  Detail & Related papers  (2022-05-25T17:59:53Z)
• AvatarMe++: Facial Shape and BRDF Inference with Photorealistic Rendering-Aware GANs [119.23922747230193]
  We introduce the first method that is able to reconstruct render-ready 3D facial geometry and BRDF from a single "in-the-wild" image. Our method outperforms the existing arts by a significant margin and reconstructs high-resolution 3D faces from a single low-resolution image.
  arXiv  Detail & Related papers  (2021-12-11T11:36:30Z)
• 3D Human Texture Estimation from a Single Image with Transformers [106.6320286821364]
  We propose a Transformer-based framework for 3D human texture estimation from a single image. We also propose a mask-fusion strategy to combine the advantages of the RGB-based and texture-flow-based models.
  arXiv  Detail & Related papers  (2021-09-06T16:00:20Z)
• SIDER: Single-Image Neural Optimization for Facial Geometric Detail Recovery [54.64663713249079]
  SIDER is a novel photometric optimization method that recovers detailed facial geometry from a single image in an unsupervised manner. In contrast to prior work, SIDER does not rely on any dataset priors and does not require additional supervision from multiple views, lighting changes or ground truth 3D shape.
  arXiv  Detail & Related papers  (2021-08-11T22:34:53Z)
• From Points to Multi-Object 3D Reconstruction [71.17445805257196]
  We propose a method to detect and reconstruct multiple 3D objects from a single RGB image. A keypoint detector localizes objects as center points and directly predicts all object properties, including 9-DoF bounding boxes and 3D shapes. The presented approach performs lightweight reconstruction in a single-stage, it is real-time capable, fully differentiable and end-to-end trainable.
  arXiv  Detail & Related papers  (2020-12-21T18:52:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.