Geometry-aware Single-image Full-body Human Relighting
- URL: http://arxiv.org/abs/2207.04750v2
- Date: Tue, 12 Jul 2022 15:08:16 GMT
- Title: Geometry-aware Single-image Full-body Human Relighting
- Authors: Chaonan Ji, Tao Yu, Kaiwen Guo, Jingxin Liu, Yebin Liu
- Abstract summary: Single-image human relighting aims to relight a target human under new lighting conditions by decomposing the input image into albedo, shape and lighting.
Previous methods suffer from both the entanglement between albedo and lighting and the lack of hard shadows.
Our framework is able to generate photo-realistic high-frequency shadows such as cast shadows under challenging lighting conditions.
- Score: 37.381122678376805
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: Single-image human relighting aims to relight a target human under new
lighting conditions by decomposing the input image into albedo, shape and
lighting. Although plausible relighting results can be achieved, previous
methods suffer from both the entanglement between albedo and lighting and the
lack of hard shadows, which significantly decrease the realism. To tackle these
two problems, we propose a geometry-aware single-image human relighting
framework that leverages single-image geometry reconstruction for joint
deployment of traditional graphics rendering and neural rendering techniques.
For the de-lighting, we explore the shortcomings of UNet architecture and
propose a modified HRNet, achieving better disentanglement between albedo and
lighting. For the relighting, we introduce a ray tracing-based per-pixel
lighting representation that explicitly models high-frequency shadows and
propose a learning-based shading refinement module to restore realistic shadows
(including hard cast shadows) from the ray-traced shading maps. Our framework
is able to generate photo-realistic high-frequency shadows such as cast shadows
under challenging lighting conditions. Extensive experiments demonstrate that
our proposed method outperforms previous methods on both synthetic and real
images.
Related papers
- Gaussian Shadow Casting for Neural Characters [20.78790953284832]
We propose a new shadow model using a Gaussian density proxy that replaces sampling with a simple analytic formula.
It supports dynamic motion and is tailored for shadow computation, thereby avoiding the affine projection approximation and sorting required by the closely related Gaussian splatting.
We demonstrate improved reconstructions, with better separation of albedo, shading, and shadows in challenging outdoor scenes with direct sun light and hard shadows.
arXiv Detail & Related papers (2024-01-11T18:50:31Z) - Relightable and Animatable Neural Avatars from Videos [14.091229306680697]
We propose a method to create relightable and animatable neural avatars.
The key challenge is to disentangle the geometry, material of the clothed body, and lighting.
Experiments on synthetic and real datasets show that our approach reconstructs high-quality geometry.
arXiv Detail & Related papers (2023-12-20T09:39:55Z) - Neural Fields meet Explicit Geometric Representation for Inverse
Rendering of Urban Scenes [62.769186261245416]
We present a novel inverse rendering framework for large urban scenes capable of jointly reconstructing the scene geometry, spatially-varying materials, and HDR lighting from a set of posed RGB images with optional depth.
Specifically, we use a neural field to account for the primary rays, and use an explicit mesh (reconstructed from the underlying neural field) for modeling secondary rays that produce higher-order lighting effects such as cast shadows.
arXiv Detail & Related papers (2023-04-06T17:51:54Z) - Learning to Relight Portrait Images via a Virtual Light Stage and
Synthetic-to-Real Adaptation [76.96499178502759]
Relighting aims to re-illuminate the person in the image as if the person appeared in an environment with the target lighting.
Recent methods rely on deep learning to achieve high-quality results.
We propose a new approach that can perform on par with the state-of-the-art (SOTA) relighting methods without requiring a light stage.
arXiv Detail & Related papers (2022-09-21T17:15:58Z) - Physically-Based Editing of Indoor Scene Lighting from a Single Image [106.60252793395104]
We present a method to edit complex indoor lighting from a single image with its predicted depth and light source segmentation masks.
We tackle this problem using two novel components: 1) a holistic scene reconstruction method that estimates scene reflectance and parametric 3D lighting, and 2) a neural rendering framework that re-renders the scene from our predictions.
arXiv Detail & Related papers (2022-05-19T06:44:37Z) - DIB-R++: Learning to Predict Lighting and Material with a Hybrid
Differentiable Renderer [78.91753256634453]
We consider the challenging problem of predicting intrinsic object properties from a single image by exploiting differentiables.
In this work, we propose DIBR++, a hybrid differentiable which supports these effects by combining specularization and ray-tracing.
Compared to more advanced physics-based differentiables, DIBR++ is highly performant due to its compact and expressive model.
arXiv Detail & Related papers (2021-10-30T01:59:39Z) - Self-supervised Outdoor Scene Relighting [92.20785788740407]
We propose a self-supervised approach for relighting.
Our approach is trained only on corpora of images collected from the internet without any user-supervision.
Results show the ability of our technique to produce photo-realistic and physically plausible results, that generalizes to unseen scenes.
arXiv Detail & Related papers (2021-07-07T09:46:19Z) - Towards High Fidelity Monocular Face Reconstruction with Rich
Reflectance using Self-supervised Learning and Ray Tracing [49.759478460828504]
Methods combining deep neural network encoders with differentiable rendering have opened up the path for very fast monocular reconstruction of geometry, lighting and reflectance.
ray tracing was introduced for monocular face reconstruction within a classic optimization-based framework.
We propose a new method that greatly improves reconstruction quality and robustness in general scenes.
arXiv Detail & Related papers (2021-03-29T08:58:10Z) - Towards Geometry Guided Neural Relighting with Flash Photography [26.511476565209026]
We propose a framework for image relighting from a single flash photograph with its corresponding depth map using deep learning.
We experimentally validate the advantage of our geometry guided approach over state-of-the-art image-based approaches in intrinsic image decomposition and image relighting.
arXiv Detail & Related papers (2020-08-12T08:03:28Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.