Face Relighting with Geometrically Consistent Shadows

• URL: http://arxiv.org/abs/2203.16681v1
• Date: Wed, 30 Mar 2022 21:31:24 GMT
• Title: Face Relighting with Geometrically Consistent Shadows
• Authors: Andrew Hou, Michel Sarkis, Ning Bi, Yiying Tong, Xiaoming Liu
• Abstract summary: We propose a novel differentiable algorithm for synthesizing hard shadows based on ray tracing. Our proposed algorithm directly utilizes the estimated face geometry to synthesize geometrically consistent hard shadows.
• Score: 24.059642361082343
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Most face relighting methods are able to handle diffuse shadows, but struggle to handle hard shadows, such as those cast by the nose. Methods that propose techniques for handling hard shadows often do not produce geometrically consistent shadows since they do not directly leverage the estimated face geometry while synthesizing them. We propose a novel differentiable algorithm for synthesizing hard shadows based on ray tracing, which we incorporate into training our face relighting model. Our proposed algorithm directly utilizes the estimated face geometry to synthesize geometrically consistent hard shadows. We demonstrate through quantitative and qualitative experiments on Multi-PIE and FFHQ that our method produces more geometrically consistent shadows than previous face relighting methods while also achieving state-of-the-art face relighting performance under directional lighting. In addition, we demonstrate that our differentiable hard shadow modeling improves the quality of the estimated face geometry over diffuse shading models.

Related papers

• Single-Image Shadow Removal Using Deep Learning: A Comprehensive Survey [78.84004293081631]
  The patterns of shadows are arbitrary, varied, and often have highly complex trace structures. The degradation caused by shadows is spatially non-uniform, resulting in inconsistencies in illumination and color between shadow and non-shadow areas. Recent developments in this field are primarily driven by deep learning-based solutions.
  arXiv  Detail & Related papers  (2024-07-11T20:58:38Z)
• Gaussian Shadow Casting for Neural Characters [20.78790953284832]
  We propose a new shadow model using a Gaussian density proxy that replaces sampling with a simple analytic formula. It supports dynamic motion and is tailored for shadow computation, thereby avoiding the affine projection approximation and sorting required by the closely related Gaussian splatting. We demonstrate improved reconstructions, with better separation of albedo, shading, and shadows in challenging outdoor scenes with direct sun light and hard shadows.
  arXiv  Detail & Related papers  (2024-01-11T18:50:31Z)
• Controllable Shadow Generation Using Pixel Height Maps [58.59256060452418]
  Physics-based shadow rendering methods require 3D geometries, which are not always available. Deep learning-based shadow synthesis methods learn a mapping from the light information to an object's shadow without explicitly modeling the shadow geometry. We introduce pixel heigh, a novel geometry representation that encodes the correlations between objects, ground, and camera pose.
  arXiv  Detail & Related papers  (2022-07-12T08:29:51Z)
• Geometry-aware Single-image Full-body Human Relighting [37.381122678376805]
  Single-image human relighting aims to relight a target human under new lighting conditions by decomposing the input image into albedo, shape and lighting. Previous methods suffer from both the entanglement between albedo and lighting and the lack of hard shadows. Our framework is able to generate photo-realistic high-frequency shadows such as cast shadows under challenging lighting conditions.
  arXiv  Detail & Related papers  (2022-07-11T10:21:02Z)
• Towards Learning Neural Representations from Shadows [11.60149896896201]
  We present a method that learns neural scene representations from only shadows present in the scene. Our framework is highly generalizable and can work alongside existing 3D reconstruction techniques.
  arXiv  Detail & Related papers  (2022-03-29T23:13:41Z)
• Neural Reflectance for Shape Recovery with Shadow Handling [88.67603644930466]
  This paper aims at recovering the shape of a scene with unknown, non-Lambertian, and possibly spatially-varying surface materials. We propose a coordinate-based deep reflectance (multilayer perceptron) to parameterize both the unknown 3D shape and the unknown at every surface point. This network is able to leverage the observed photometric variance and shadows on the surface, and recover both surface shape and general non-Lambertian reflectance.
  arXiv  Detail & Related papers  (2022-03-24T07:57:20Z)
• SIDER: Single-Image Neural Optimization for Facial Geometric Detail Recovery [54.64663713249079]
  SIDER is a novel photometric optimization method that recovers detailed facial geometry from a single image in an unsupervised manner. In contrast to prior work, SIDER does not rely on any dataset priors and does not require additional supervision from multiple views, lighting changes or ground truth 3D shape.
  arXiv  Detail & Related papers  (2021-08-11T22:34:53Z)
• Towards High Fidelity Face Relighting with Realistic Shadows [21.09340135707926]
  Our method learns to predict the ratio (quotient) image between a source image and the target image with the desired lighting. During training, our model also learns to accurately modify shadows by using estimated shadow masks. We demonstrate that our proposed method faithfully maintains the local facial details of the subject and can accurately handle hard shadows.
  arXiv  Detail & Related papers  (2021-04-02T00:28:40Z)
• Efficient and Differentiable Shadow Computation for Inverse Problems [64.70468076488419]
  Differentiable geometric computation has received increasing interest for image-based inverse problems. We propose an efficient yet efficient approach for differentiable visibility and soft shadow computation. As our formulation is differentiable, it can be used to solve inverse problems such as texture, illumination, rigid pose, and deformation recovery from images.
  arXiv  Detail & Related papers  (2021-04-01T09:29:05Z)
• Towards High Fidelity Monocular Face Reconstruction with Rich Reflectance using Self-supervised Learning and Ray Tracing [49.759478460828504]
  Methods combining deep neural network encoders with differentiable rendering have opened up the path for very fast monocular reconstruction of geometry, lighting and reflectance. ray tracing was introduced for monocular face reconstruction within a classic optimization-based framework. We propose a new method that greatly improves reconstruction quality and robustness in general scenes.
  arXiv  Detail & Related papers  (2021-03-29T08:58:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.