Towards High Fidelity Face Relighting with Realistic Shadows

• URL: http://arxiv.org/abs/2104.00825v1
• Date: Fri, 2 Apr 2021 00:28:40 GMT
• Title: Towards High Fidelity Face Relighting with Realistic Shadows
• Authors: Andrew Hou, Ze Zhang, Michel Sarkis, Ning Bi, Yiying Tong, Xiaoming Liu
• Abstract summary: Our method learns to predict the ratio (quotient) image between a source image and the target image with the desired lighting. During training, our model also learns to accurately modify shadows by using estimated shadow masks. We demonstrate that our proposed method faithfully maintains the local facial details of the subject and can accurately handle hard shadows.
• Score: 21.09340135707926
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing face relighting methods often struggle with two problems: maintaining the local facial details of the subject and accurately removing and synthesizing shadows in the relit image, especially hard shadows. We propose a novel deep face relighting method that addresses both problems. Our method learns to predict the ratio (quotient) image between a source image and the target image with the desired lighting, allowing us to relight the image while maintaining the local facial details. During training, our model also learns to accurately modify shadows by using estimated shadow masks to emphasize on the high-contrast shadow borders. Furthermore, we introduce a method to use the shadow mask to estimate the ambient light intensity in an image, and are thus able to leverage multiple datasets during training with different global lighting intensities. With quantitative and qualitative evaluations on the Multi-PIE and FFHQ datasets, we demonstrate that our proposed method faithfully maintains the local facial details of the subject and can accurately handle hard shadows while achieving state-of-the-art face relighting performance.

Related papers

• Single-Image Shadow Removal Using Deep Learning: A Comprehensive Survey [78.84004293081631]
  The patterns of shadows are arbitrary, varied, and often have highly complex trace structures. The degradation caused by shadows is spatially non-uniform, resulting in inconsistencies in illumination and color between shadow and non-shadow areas. Recent developments in this field are primarily driven by deep learning-based solutions.
  arXiv  Detail & Related papers  (2024-07-11T20:58:38Z)
• SIRe-IR: Inverse Rendering for BRDF Reconstruction with Shadow and Illumination Removal in High-Illuminance Scenes [51.50157919750782]
  We present SIRe-IR, an implicit neural rendering inverse approach that decomposes the scene into environment map, albedo, and roughness. By accurately modeling the indirect radiance field, normal, visibility, and direct light simultaneously, we are able to remove both shadows and indirect illumination. Even in the presence of intense illumination, our method recovers high-quality albedo and roughness with no shadow interference.
  arXiv  Detail & Related papers  (2023-10-19T10:44:23Z)
• Controllable Light Diffusion for Portraits [8.931046902694984]
  We introduce light diffusion, a novel method to improve lighting in portraits. Inspired by professional photographers' diffusers and scrims, our method softens lighting given only a single portrait photo.
  arXiv  Detail & Related papers  (2023-05-08T14:46:28Z)
• Face Relighting with Geometrically Consistent Shadows [24.059642361082343]
  We propose a novel differentiable algorithm for synthesizing hard shadows based on ray tracing. Our proposed algorithm directly utilizes the estimated face geometry to synthesize geometrically consistent hard shadows.
  arXiv  Detail & Related papers  (2022-03-30T21:31:24Z)
• Deep Portrait Delighting [9.455161801306739]
  We present a deep neural network for removing undesirable shading features from an unconstrained portrait image. Our training scheme incorporates three regularization strategies: masked loss, to emphasize high-frequency shading features; soft-shadow loss, which improves sensitivity to subtle changes in lighting. Our method demonstrates improved delighting quality and generalization when compared with the state-of-the-art.
  arXiv  Detail & Related papers  (2022-03-22T22:51:22Z)
• Self-Supervised Shadow Removal [130.6657167667636]
  We propose an unsupervised single image shadow removal solution via self-supervised learning by using a conditioned mask. In contrast to existing literature, we do not require paired shadowed and shadow-free images, instead we rely on self-supervision and jointly learn deep models to remove and add shadows to images.
  arXiv  Detail & Related papers  (2020-10-22T11:33:41Z)
• Learning Flow-based Feature Warping for Face Frontalization with Illumination Inconsistent Supervision [73.18554605744842]
  Flow-based Feature Warping Model (FFWM) learns to synthesize photo-realistic and illumination preserving frontal images. An Illumination Preserving Module (IPM) is proposed to learn illumination preserving image synthesis. A Warp Attention Module (WAM) is introduced to reduce the pose discrepancy in the feature level.
  arXiv  Detail & Related papers  (2020-08-16T06:07:00Z)
• Towards Geometry Guided Neural Relighting with Flash Photography [26.511476565209026]
  We propose a framework for image relighting from a single flash photograph with its corresponding depth map using deep learning. We experimentally validate the advantage of our geometry guided approach over state-of-the-art image-based approaches in intrinsic image decomposition and image relighting.
  arXiv  Detail & Related papers  (2020-08-12T08:03:28Z)
• Deep Reflectance Volumes: Relightable Reconstructions from Multi-View Photometric Images [59.53382863519189]
  We present a deep learning approach to reconstruct scene appearance from unstructured images captured under collocated point lighting. At the heart of Deep Reflectance Volumes is a novel volumetric scene representation consisting of opacity, surface normal and reflectance voxel grids. We show that our learned reflectance volumes are editable, allowing for modifying the materials of the captured scenes.
  arXiv  Detail & Related papers  (2020-07-20T05:38:11Z)
• Deep Bilateral Retinex for Low-Light Image Enhancement [96.15991198417552]
  Low-light images suffer from poor visibility caused by low contrast, color distortion and measurement noise. This paper proposes a deep learning method for low-light image enhancement with a particular focus on handling the measurement noise. The proposed method is very competitive to the state-of-the-art methods, and has significant advantage over others when processing images captured in extremely low lighting conditions.
  arXiv  Detail & Related papers  (2020-07-04T06:26:44Z)
• Portrait Shadow Manipulation [37.414681268753526]
  Casually-taken portrait photographs often suffer from unflattering lighting and shadowing because of suboptimal conditions in the environment. We present a computational approach that gives casual photographers some of this control, thereby allowing poorly-lit portraits to be relit post-capture in a realistic and easily-controllable way. Our approach relies on a pair of neural networks---one to remove foreign shadows cast by external objects, and another to soften facial shadows cast by the features of the subject and to add a synthetic fill light to improve the lighting ratio.
  arXiv  Detail & Related papers  (2020-05-18T17:51:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.