Single-image Full-body Human Relighting
- URL: http://arxiv.org/abs/2107.07259v1
- Date: Thu, 15 Jul 2021 11:34:03 GMT
- Title: Single-image Full-body Human Relighting
- Authors: Manuel Lagunas, Xin Sun, Jimei Yang, Ruben Villegas, Jianming Zhang,
Zhixin Shu, Belen Masia, and Diego Gutierrez
- Abstract summary: We present a single-image data-driven method to automatically relight images with full-body humans in them.
Our framework is based on a realistic scene decomposition leveraging precomputed radiance transfer (PRT) and spherical harmonics (SH) lighting.
We propose a new deep learning architecture, tailored to the decomposition performed in PRT, that is trained using a combination of L1, logarithmic, and rendering losses.
- Score: 42.06323641073984
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: We present a single-image data-driven method to automatically relight images
with full-body humans in them. Our framework is based on a realistic scene
decomposition leveraging precomputed radiance transfer (PRT) and spherical
harmonics (SH) lighting. In contrast to previous work, we lift the assumptions
on Lambertian materials and explicitly model diffuse and specular reflectance
in our data. Moreover, we introduce an additional light-dependent residual term
that accounts for errors in the PRT-based image reconstruction. We propose a
new deep learning architecture, tailored to the decomposition performed in PRT,
that is trained using a combination of L1, logarithmic, and rendering losses.
Our model outperforms the state of the art for full-body human relighting both
with synthetic images and photographs.
Related papers
- TensoIR: Tensorial Inverse Rendering [51.57268311847087]
TensoIR is a novel inverse rendering approach based on tensor factorization and neural fields.
TensoRF is a state-of-the-art approach for radiance field modeling.
arXiv Detail & Related papers (2023-04-24T21:39:13Z) - Neural Radiance Transfer Fields for Relightable Novel-view Synthesis
with Global Illumination [63.992213016011235]
We propose a method for scene relighting under novel views by learning a neural precomputed radiance transfer function.
Our method can be solely supervised on a set of real images of the scene under a single unknown lighting condition.
Results show that the recovered disentanglement of scene parameters improves significantly over the current state of the art.
arXiv Detail & Related papers (2022-07-27T16:07:48Z) - Contrastive Feature Loss for Image Prediction [55.373404869092866]
Training supervised image synthesis models requires a critic to compare two images: the ground truth to the result.
We introduce an information theory based approach to measuring similarity between two images.
We show that our formulation boosts the perceptual realism of output images when used as a drop-in replacement for the L1 loss.
arXiv Detail & Related papers (2021-11-12T20:39:52Z) - DIB-R++: Learning to Predict Lighting and Material with a Hybrid
Differentiable Renderer [78.91753256634453]
We consider the challenging problem of predicting intrinsic object properties from a single image by exploiting differentiables.
In this work, we propose DIBR++, a hybrid differentiable which supports these effects by combining specularization and ray-tracing.
Compared to more advanced physics-based differentiables, DIBR++ is highly performant due to its compact and expressive model.
arXiv Detail & Related papers (2021-10-30T01:59:39Z) - DA-DRN: Degradation-Aware Deep Retinex Network for Low-Light Image
Enhancement [14.75902042351609]
We propose a Degradation-Aware Deep Retinex Network (denoted as DA-DRN) for low-light image enhancement and tackle the above degradation.
Based on Retinex Theory, the decomposition net in our model can decompose low-light images into reflectance and illumination maps.
We conduct extensive experiments to demonstrate that our approach achieves a promising effect with good rubustness and generalization.
arXiv Detail & Related papers (2021-10-05T03:53:52Z) - Intrinsic Image Transfer for Illumination Manipulation [1.2387676601792899]
This paper presents a novel intrinsic image transfer (IIT) algorithm for illumination manipulation.
It creates a local image translation between two illumination surfaces.
We illustrate that all losses can be reduced without the necessity of taking an intrinsic image decomposition.
arXiv Detail & Related papers (2021-07-01T19:12:24Z) - Joint Learning of Portrait Intrinsic Decomposition and Relighting [5.601217969637838]
Inverse rendering is the problem of decomposing an image into its intrinsic components, i.e. albedo, normal and lighting.
Here, we propose a new self-supervised training paradigm that reduces the need for full supervision on the decomposition task.
We show-case the effectiveness of our training paradigm on both intrinsic decomposition and relighting and demonstrate how the model struggles in both tasks without the self-supervised loss terms.
arXiv Detail & Related papers (2021-06-22T19:40:22Z) - Neural Light Transport for Relighting and View Synthesis [70.39907425114302]
Light transport (LT) of a scene describes how it appears under different lighting and viewing directions.
We propose a semi-parametric approach to learn a neural representation of LT embedded in a texture atlas of known geometric properties.
We show how to fuse previously seen observations of illuminants and views to synthesize a new image of the same scene under a desired lighting condition.
arXiv Detail & Related papers (2020-08-09T20:13:15Z) - Deep Lighting Environment Map Estimation from Spherical Panoramas [0.0]
We present a data-driven model that estimates an HDR lighting environment map from a single LDR monocular spherical panorama.
We exploit the availability of surface geometry to employ image-based relighting as a data generator and supervision mechanism.
arXiv Detail & Related papers (2020-05-16T14:23:05Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.