HairStep: Transfer Synthetic to Real Using Strand and Depth Maps for Single-View 3D Hair Modeling

• URL: http://arxiv.org/abs/2303.02700v2
• Date: Fri, 24 Mar 2023 03:34:25 GMT
• Title: HairStep: Transfer Synthetic to Real Using Strand and Depth Maps for Single-View 3D Hair Modeling
• Authors: Yujian Zheng, Zirong Jin, Moran Li, Haibin Huang, Chongyang Ma, Shuguang Cui, Xiaoguang Han
• Abstract summary: We tackle the challenging problem of learning-based single-view 3D hair modeling. We first propose a novel intermediate representation, termed as HairStep, which consists of a strand map and a depth map. It is found that HairStep not only provides sufficient information for accurate 3D hair modeling, but also is feasible to be inferred from real images.
• Score: 55.57803336895614
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this work, we tackle the challenging problem of learning-based single-view 3D hair modeling. Due to the great difficulty of collecting paired real image and 3D hair data, using synthetic data to provide prior knowledge for real domain becomes a leading solution. This unfortunately introduces the challenge of domain gap. Due to the inherent difficulty of realistic hair rendering, existing methods typically use orientation maps instead of hair images as input to bridge the gap. We firmly think an intermediate representation is essential, but we argue that orientation map using the dominant filtering-based methods is sensitive to uncertain noise and far from a competent representation. Thus, we first raise this issue up and propose a novel intermediate representation, termed as HairStep, which consists of a strand map and a depth map. It is found that HairStep not only provides sufficient information for accurate 3D hair modeling, but also is feasible to be inferred from real images. Specifically, we collect a dataset of 1,250 portrait images with two types of annotations. A learning framework is further designed to transfer real images to the strand map and depth map. It is noted that, an extra bonus of our new dataset is the first quantitative metric for 3D hair modeling. Our experiments show that HairStep narrows the domain gap between synthetic and real and achieves state-of-the-art performance on single-view 3D hair reconstruction.

Related papers

• Perm: A Parametric Representation for Multi-Style 3D Hair Modeling [22.790597419351528]
  Perm is a learned parametric model of human 3D hair designed to facilitate various hair-related applications. We propose to disentangle the global hair shape and local strand details using a PCA-based strand representation in the frequency domain. These textures are later parameterized with different generative models, emulating common stages in the hair modeling process.
  arXiv  Detail & Related papers  (2024-07-28T10:05:11Z)
• MonoHair: High-Fidelity Hair Modeling from a Monocular Video [40.27026803872373]
  MonoHair is a generic framework to achieve high-fidelity hair reconstruction from a monocular video. Our approach bifurcates the hair modeling process into two main stages: precise exterior reconstruction and interior structure inference. Our experiments demonstrate that our method exhibits robustness across diverse hairstyles and achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-03-27T08:48:47Z)
• DI-Net : Decomposed Implicit Garment Transfer Network for Digital Clothed 3D Human [75.45488434002898]
  Existing 2D virtual try-on methods cannot be directly extended to 3D since they lack the ability to perceive the depth of each pixel. We propose a Decomposed Implicit garment transfer network (DI-Net), which can effortlessly reconstruct a 3D human mesh with the newly try-on result.
  arXiv  Detail & Related papers  (2023-11-28T14:28:41Z)
• HQ3DAvatar: High Quality Controllable 3D Head Avatar [65.70885416855782]
  This paper presents a novel approach to building highly photorealistic digital head avatars. Our method learns a canonical space via an implicit function parameterized by a neural network. At test time, our method is driven by a monocular RGB video.
  arXiv  Detail & Related papers  (2023-03-25T13:56:33Z)
• RaBit: Parametric Modeling of 3D Biped Cartoon Characters with a Topological-consistent Dataset [19.494054103293116]
  We introduce 3DBiCar, the first large-scale dataset of 3D biped cartoon characters, and RaBit, the corresponding parametric model. RaBit is built with a SMPL-like linear blend shape model and a StyleGAN-based neural UV-texture generator, simultaneously expressing the shape, pose, and texture. Various applications are conducted, including single-view reconstruction, sketch-based modeling, and 3D cartoon animation.
  arXiv  Detail & Related papers  (2023-03-22T13:46:15Z)
• Shape, Pose, and Appearance from a Single Image via Bootstrapped Radiance Field Inversion [54.151979979158085]
  We introduce a principled end-to-end reconstruction framework for natural images, where accurate ground-truth poses are not available. We leverage an unconditional 3D-aware generator, to which we apply a hybrid inversion scheme where a model produces a first guess of the solution. Our framework can de-render an image in as few as 10 steps, enabling its use in practical scenarios.
  arXiv  Detail & Related papers  (2022-11-21T17:42:42Z)
• DRaCoN -- Differentiable Rasterization Conditioned Neural Radiance Fields for Articulated Avatars [92.37436369781692]
  We present DRaCoN, a framework for learning full-body volumetric avatars. It exploits the advantages of both the 2D and 3D neural rendering techniques. Experiments on the challenging ZJU-MoCap and Human3.6M datasets indicate that DRaCoN outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-03-29T17:59:15Z)
• Neural Hair Rendering [41.25606756188364]
  We propose a generic neural-based hair rendering pipeline that can synthesize photo-realistic images from virtual 3D hair models. Key component of our method is a shared latent space to encode appearance-invariant structure information of both domains.
  arXiv  Detail & Related papers  (2020-04-28T04:36:49Z)
• HandVoxNet: Deep Voxel-Based Network for 3D Hand Shape and Pose Estimation from a Single Depth Map [72.93634777578336]
  We propose a novel architecture with 3D convolutions trained in a weakly-supervised manner. The proposed approach improves over the state of the art by 47.8% on the SynHand5M dataset. Our method produces visually more reasonable and realistic hand shapes on NYU and BigHand2.2M datasets.
  arXiv  Detail & Related papers  (2020-04-03T14:27:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.