Face Inverse Rendering via Hierarchical Decoupling

• URL: http://arxiv.org/abs/2301.06733v1
• Date: Tue, 17 Jan 2023 07:24:47 GMT
• Title: Face Inverse Rendering via Hierarchical Decoupling
• Authors: Meng Wang, Xiaojie Guo, Wenjing Dai, and Jiawan Zhang
• Abstract summary: Previous face inverse rendering methods often require synthetic data with ground truth and/or professional equipment like a lighting stage. We propose a deep learning framework to disentangle face images in the wild into their corresponding albedo, normal, and lighting components.
• Score: 19.530753479268384
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Previous face inverse rendering methods often require synthetic data with ground truth and/or professional equipment like a lighting stage. However, a model trained on synthetic data or using pre-defined lighting priors is typically unable to generalize well for real-world situations, due to the gap between synthetic data/lighting priors and real data. Furthermore, for common users, the professional equipment and skill make the task expensive and complex. In this paper, we propose a deep learning framework to disentangle face images in the wild into their corresponding albedo, normal, and lighting components. Specifically, a decomposition network is built with a hierarchical subdivision strategy, which takes image pairs captured from arbitrary viewpoints as input. In this way, our approach can greatly mitigate the pressure from data preparation, and significantly broaden the applicability of face inverse rendering. Extensive experiments are conducted to demonstrate the efficacy of our design, and show its superior performance in face relighting over other state-of-the-art alternatives. {Our code is available at \url{https://github.com/AutoHDR/HD-Net.git}}

Related papers

• A General Albedo Recovery Approach for Aerial Photogrammetric Images through Inverse Rendering [7.874736360019618]
  This paper presents a general image formation model for albedo recovery from typical aerial photogrammetric images under natural illuminations. Our approach builds on the fact that both the sun illumination and scene geometry are estimable in aerial photogrammetry.
  arXiv  Detail & Related papers  (2024-09-04T18:58:32Z)
• DiFaReli: Diffusion Face Relighting [13.000032155650835]
  We present a novel approach to single-view face relighting in the wild. Handling non-diffuse effects, such as global illumination or cast shadows, has long been a challenge in face relighting. We achieve state-of-the-art performance on standard benchmark Multi-PIE and can photorealistically relight in-the-wild images.
  arXiv  Detail & Related papers  (2023-04-19T08:03:20Z)
• Learning to Relight Portrait Images via a Virtual Light Stage and Synthetic-to-Real Adaptation [76.96499178502759]
  Relighting aims to re-illuminate the person in the image as if the person appeared in an environment with the target lighting. Recent methods rely on deep learning to achieve high-quality results. We propose a new approach that can perform on par with the state-of-the-art (SOTA) relighting methods without requiring a light stage.
  arXiv  Detail & Related papers  (2022-09-21T17:15:58Z)
• Designing An Illumination-Aware Network for Deep Image Relighting [69.750906769976]
  We present an Illumination-Aware Network (IAN) which follows the guidance from hierarchical sampling to progressively relight a scene from a single image. In addition, an Illumination-Aware Residual Block (IARB) is designed to approximate the physical rendering process. Experimental results show that our proposed method produces better quantitative and qualitative relighting results than previous state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-07-21T16:21:24Z)
• Single-image Full-body Human Relighting [42.06323641073984]
  We present a single-image data-driven method to automatically relight images with full-body humans in them. Our framework is based on a realistic scene decomposition leveraging precomputed radiance transfer (PRT) and spherical harmonics (SH) lighting. We propose a new deep learning architecture, tailored to the decomposition performed in PRT, that is trained using a combination of L1, logarithmic, and rendering losses.
  arXiv  Detail & Related papers  (2021-07-15T11:34:03Z)
• Self-supervised Outdoor Scene Relighting [92.20785788740407]
  We propose a self-supervised approach for relighting. Our approach is trained only on corpora of images collected from the internet without any user-supervision. Results show the ability of our technique to produce photo-realistic and physically plausible results, that generalizes to unseen scenes.
  arXiv  Detail & Related papers  (2021-07-07T09:46:19Z)
• Intrinsic Decomposition of Document Images In-the-Wild [28.677728405031782]
  We present a learning-based method that directly estimates document reflectance based on intrinsic image formation. The proposed architecture works in a self-supervised manner where only the synthetic texture is used as a weak training signal. Our reflectance estimation scheme, when used as a pre-processing step of an OCR pipeline, shows a 26% improvement of character error rate.
  arXiv  Detail & Related papers  (2020-11-29T21:39:58Z)
• Light Stage Super-Resolution: Continuous High-Frequency Relighting [58.09243542908402]
  We propose a learning-based solution for the "super-resolution" of scans of human faces taken from a light stage. Our method aggregates the captured images corresponding to neighboring lights in the stage, and uses a neural network to synthesize a rendering of the face. Our learned model is able to produce renderings for arbitrary light directions that exhibit realistic shadows and specular highlights.
  arXiv  Detail & Related papers  (2020-10-17T23:40:43Z)
• Automated Synthetic-to-Real Generalization [142.41531132965585]
  We propose a textitlearning-to-optimize (L2O) strategy to automate the selection of layer-wise learning rates. We demonstrate that the proposed framework can significantly improve the synthetic-to-real generalization performance without seeing and training on real data.
  arXiv  Detail & Related papers  (2020-07-14T10:57:34Z)
• Learning Inverse Rendering of Faces from Real-world Videos [52.313931830408386]
  Existing methods decompose a face image into three components (albedo, normal, and illumination) by supervised training on synthetic data. We propose a weakly supervised training approach to train our model on real face videos, based on the assumption of consistency of albedo and normal. Our network is trained on both real and synthetic data, benefiting from both.
  arXiv  Detail & Related papers  (2020-03-26T17:26:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.