Deep Lighting Environment Map Estimation from Spherical Panoramas

• URL: http://arxiv.org/abs/2005.08000v1
• Date: Sat, 16 May 2020 14:23:05 GMT
• Title: Deep Lighting Environment Map Estimation from Spherical Panoramas
• Authors: Vasileios Gkitsas (1) and Nikolaos Zioulis (1 and 2) and Federico Alvarez (2) and Dimitrios Zarpalas (1) and Petros Daras (1) ((1) Centre for Research and Technology Hellas, (2) Universidad Politecnica de Madrid)
• Abstract summary: We present a data-driven model that estimates an HDR lighting environment map from a single LDR monocular spherical panorama. We exploit the availability of surface geometry to employ image-based relighting as a data generator and supervision mechanism.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Estimating a scene's lighting is a very important task when compositing synthetic content within real environments, with applications in mixed reality and post-production. In this work we present a data-driven model that estimates an HDR lighting environment map from a single LDR monocular spherical panorama. In addition to being a challenging and ill-posed problem, the lighting estimation task also suffers from a lack of facile illumination ground truth data, a fact that hinders the applicability of data-driven methods. We approach this problem differently, exploiting the availability of surface geometry to employ image-based relighting as a data generator and supervision mechanism. This relies on a global Lambertian assumption that helps us overcome issues related to pre-baked lighting. We relight our training data and complement the model's supervision with a photometric loss, enabled by a differentiable image-based relighting technique. Finally, since we predict spherical spectral coefficients, we show that by imposing a distribution prior on the predicted coefficients, we can greatly boost performance. Code and models available at https://vcl3d.github.io/DeepPanoramaLighting.

Related papers

• Physically-Based Photometric Bundle Adjustment in Non-Lambertian Environments [59.96101889715997]
  Photometric bundle adjustment (PBA) is widely used in estimating the camera pose and 3D geometry by assuming a Lambertian world. The assumption of photometric consistency is often violated since the non-diffuse reflection is common in real-world environments. We propose a novel physically-based PBA method to solve this problem.
  arXiv  Detail & Related papers  (2024-09-18T10:22:07Z)
• SplitNeRF: Split Sum Approximation Neural Field for Joint Geometry, Illumination, and Material Estimation [65.99344783327054]
  We present a novel approach for digitizing real-world objects by estimating their geometry, material properties, and lighting. Our method incorporates into Radiance Neural Field (NeRF) pipelines the split sum approximation used with image-based lighting for real-time physical-based rendering. Our method is capable of attaining state-of-the-art relighting quality after only $sim1$ hour of training in a single NVIDIA A100 GPU.
  arXiv  Detail & Related papers  (2023-11-28T10:36:36Z)
• Spatiotemporally Consistent HDR Indoor Lighting Estimation [66.26786775252592]
  We propose a physically-motivated deep learning framework to solve the indoor lighting estimation problem. Given a single LDR image with a depth map, our method predicts spatially consistent lighting at any given image position. Our framework achieves photorealistic lighting prediction with higher quality compared to state-of-the-art single-image or video-based methods.
  arXiv  Detail & Related papers  (2023-05-07T20:36:29Z)
• EverLight: Indoor-Outdoor Editable HDR Lighting Estimation [9.443561684223514]
  We propose a method which combines a parametric light model with 360deg panoramas, ready to use as HDRI in rendering engines. In our representation, users can easily edit light direction, intensity, number, etc. to impact shading while providing rich, complex reflections while seamlessly blending with the edits.
  arXiv  Detail & Related papers  (2023-04-26T00:20:59Z)
• A CNN Based Approach for the Point-Light Photometric Stereo Problem [26.958763133729846]
  We propose a CNN-based approach capable of handling realistic assumptions by leveraging recent improvements of deep neural networks for far-field Photometric Stereo. Our approach outperforms the state-of-the-art on the DiLiGenT real world dataset. In order to measure the performance of our approach for near-field point-light source PS data, we introduce LUCES the first real-world 'dataset for near-fieLd point light soUrCe photomEtric Stereo'
  arXiv  Detail & Related papers  (2022-10-10T12:57:12Z)
• Neural Light Field Estimation for Street Scenes with Differentiable Virtual Object Insertion [129.52943959497665]
  Existing works on outdoor lighting estimation typically simplify the scene lighting into an environment map. We propose a neural approach that estimates the 5D HDR light field from a single image. We show the benefits of our AR object insertion in an autonomous driving application.
  arXiv  Detail & Related papers  (2022-08-19T17:59:16Z)
• DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderer [78.91753256634453]
  We consider the challenging problem of predicting intrinsic object properties from a single image by exploiting differentiables. In this work, we propose DIBR++, a hybrid differentiable which supports these effects by combining specularization and ray-tracing. Compared to more advanced physics-based differentiables, DIBR++ is highly performant due to its compact and expressive model.
  arXiv  Detail & Related papers  (2021-10-30T01:59:39Z)
• Spatially and color consistent environment lighting estimation using deep neural networks for mixed reality [1.1470070927586016]
  This paper presents a CNN-based model to estimate complex lighting for mixed reality environments. We propose a new CNN architecture that inputs an RGB image and recognizes, in real-time, the environment lighting. We show in experiments that the CNN architecture can predict the environment lighting with an average mean squared error (MSE) of num7.85e-04 when comparing SH lighting coefficients.
  arXiv  Detail & Related papers  (2021-08-17T23:03:55Z)
• GMLight: Lighting Estimation via Geometric Distribution Approximation [86.95367898017358]
  This paper presents a lighting estimation framework that employs a regression network and a generative projector for effective illumination estimation. We parameterize illumination scenes in terms of the geometric light distribution, light intensity, ambient term, and auxiliary depth, and estimate them as a pure regression task. With the estimated lighting parameters, the generative projector synthesizes panoramic illumination maps with realistic appearance and frequency.
  arXiv  Detail & Related papers  (2021-02-20T03:31:52Z)
• PointAR: Efficient Lighting Estimation for Mobile Augmented Reality [7.58114840374767]
  We propose an efficient lighting estimation pipeline that is suitable to run on modern mobile devices. PointAR takes a single RGB-D image captured from the mobile camera and a 2D location in that image, and estimates 2nd order spherical harmonics coefficients.
  arXiv  Detail & Related papers  (2020-03-30T19:13:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.