Spatiotemporally Consistent HDR Indoor Lighting Estimation
- URL: http://arxiv.org/abs/2305.04374v1
- Date: Sun, 7 May 2023 20:36:29 GMT
- Title: Spatiotemporally Consistent HDR Indoor Lighting Estimation
- Authors: Zhengqin Li, Li Yu, Mikhail Okunev, Manmohan Chandraker, Zhao Dong
- Abstract summary: We propose a physically-motivated deep learning framework to solve the indoor lighting estimation problem.
Given a single LDR image with a depth map, our method predicts spatially consistent lighting at any given image position.
Our framework achieves photorealistic lighting prediction with higher quality compared to state-of-the-art single-image or video-based methods.
- Score: 66.26786775252592
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: We propose a physically-motivated deep learning framework to solve a general
version of the challenging indoor lighting estimation problem. Given a single
LDR image with a depth map, our method predicts spatially consistent lighting
at any given image position. Particularly, when the input is an LDR video
sequence, our framework not only progressively refines the lighting prediction
as it sees more regions, but also preserves temporal consistency by keeping the
refinement smooth. Our framework reconstructs a spherical Gaussian lighting
volume (SGLV) through a tailored 3D encoder-decoder, which enables spatially
consistent lighting prediction through volume ray tracing, a hybrid blending
network for detailed environment maps, an in-network Monte-Carlo rendering
layer to enhance photorealism for virtual object insertion, and recurrent
neural networks (RNN) to achieve temporally consistent lighting prediction with
a video sequence as the input. For training, we significantly enhance the
OpenRooms public dataset of photorealistic synthetic indoor scenes with around
360K HDR environment maps of much higher resolution and 38K video sequences,
rendered with GPU-based path tracing. Experiments show that our framework
achieves lighting prediction with higher quality compared to state-of-the-art
single-image or video-based methods, leading to photorealistic AR applications
such as object insertion.
Related papers
- PBIR-NIE: Glossy Object Capture under Non-Distant Lighting [30.325872237020395]
Glossy objects present a significant challenge for 3D reconstruction from multi-view input images under natural lighting.
We introduce PBIR-NIE, an inverse rendering framework designed to holistically capture the geometry, material attributes, and surrounding illumination of such objects.
arXiv Detail & Related papers (2024-08-13T13:26:24Z) - Relighting Scenes with Object Insertions in Neural Radiance Fields [24.18050535794117]
We propose a novel NeRF-based pipeline for inserting object NeRFs into scene NeRFs.
The proposed method achieves realistic relighting effects in extensive experimental evaluations.
arXiv Detail & Related papers (2024-06-21T00:58:58Z) - Relightable 3D Gaussians: Realistic Point Cloud Relighting with BRDF Decomposition and Ray Tracing [21.498078188364566]
We present a novel differentiable point-based rendering framework to achieve photo-realistic relighting.
The proposed framework showcases the potential to revolutionize the mesh-based graphics pipeline with a point-based pipeline enabling editing, tracing, and relighting.
arXiv Detail & Related papers (2023-11-27T18:07:58Z) - Neural Fields meet Explicit Geometric Representation for Inverse
Rendering of Urban Scenes [62.769186261245416]
We present a novel inverse rendering framework for large urban scenes capable of jointly reconstructing the scene geometry, spatially-varying materials, and HDR lighting from a set of posed RGB images with optional depth.
Specifically, we use a neural field to account for the primary rays, and use an explicit mesh (reconstructed from the underlying neural field) for modeling secondary rays that produce higher-order lighting effects such as cast shadows.
arXiv Detail & Related papers (2023-04-06T17:51:54Z) - Learning-based Inverse Rendering of Complex Indoor Scenes with
Differentiable Monte Carlo Raytracing [27.96634370355241]
This work presents an end-to-end, learning-based inverse rendering framework incorporating differentiable Monte Carlo raytracing with importance sampling.
The framework takes a single image as input to jointly recover the underlying geometry, spatially-varying lighting, and photorealistic materials.
arXiv Detail & Related papers (2022-11-06T03:34:26Z) - Neural Light Field Estimation for Street Scenes with Differentiable
Virtual Object Insertion [129.52943959497665]
Existing works on outdoor lighting estimation typically simplify the scene lighting into an environment map.
We propose a neural approach that estimates the 5D HDR light field from a single image.
We show the benefits of our AR object insertion in an autonomous driving application.
arXiv Detail & Related papers (2022-08-19T17:59:16Z) - Deep Graph Learning for Spatially-Varying Indoor Lighting Prediction [25.726519831985918]
We propose a graph learning-based framework for indoor lighting estimation.
At its core is a new lighting model (dubbed DSGLight) based on depth-augmented Spherical Gaussians.
Our method obviously outperforms existing methods both qualitatively and quantitatively.
arXiv Detail & Related papers (2022-02-13T12:49:37Z) - Learning Indoor Inverse Rendering with 3D Spatially-Varying Lighting [149.1673041605155]
We address the problem of jointly estimating albedo, normals, depth and 3D spatially-varying lighting from a single image.
Most existing methods formulate the task as image-to-image translation, ignoring the 3D properties of the scene.
We propose a unified, learning-based inverse framework that formulates 3D spatially-varying lighting.
arXiv Detail & Related papers (2021-09-13T15:29:03Z) - Neural Reflectance Fields for Appearance Acquisition [61.542001266380375]
We present Neural Reflectance Fields, a novel deep scene representation that encodes volume density, normal and reflectance properties at any 3D point in a scene.
We combine this representation with a physically-based differentiable ray marching framework that can render images from a neural reflectance field under any viewpoint and light.
arXiv Detail & Related papers (2020-08-09T22:04:36Z) - Deep 3D Capture: Geometry and Reflectance from Sparse Multi-View Images [59.906948203578544]
We introduce a novel learning-based method to reconstruct the high-quality geometry and complex, spatially-varying BRDF of an arbitrary object.
We first estimate per-view depth maps using a deep multi-view stereo network.
These depth maps are used to coarsely align the different views.
We propose a novel multi-view reflectance estimation network architecture.
arXiv Detail & Related papers (2020-03-27T21:28:54Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.