Neural Radiance Fields for Outdoor Scene Relighting

• URL: http://arxiv.org/abs/2112.05140v1
• Date: Thu, 9 Dec 2021 18:59:56 GMT
• Title: Neural Radiance Fields for Outdoor Scene Relighting
• Authors: Viktor Rudnev and Mohamed Elgharib and William Smith and Lingjie Liu and Vladislav Golyanik and Christian Theobalt
• Abstract summary: We present NeRF-OSR, the first approach for outdoor scene relighting based on neural radiance fields. In contrast to the prior art, our technique allows simultaneous editing of both scene illumination and camera viewpoint. It also includes a dedicated network for shadow reproduction, which is crucial for high-quality outdoor scene relighting.
• Score: 70.97747511934705
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Photorealistic editing of outdoor scenes from photographs requires a profound understanding of the image formation process and an accurate estimation of the scene geometry, reflectance and illumination. A delicate manipulation of the lighting can then be performed while keeping the scene albedo and geometry unaltered. We present NeRF-OSR, i.e., the first approach for outdoor scene relighting based on neural radiance fields. In contrast to the prior art, our technique allows simultaneous editing of both scene illumination and camera viewpoint using only a collection of outdoor photos shot in uncontrolled settings. Moreover, it enables direct control over the scene illumination, as defined through a spherical harmonics model. It also includes a dedicated network for shadow reproduction, which is crucial for high-quality outdoor scene relighting. To evaluate the proposed method, we collect a new benchmark dataset of several outdoor sites, where each site is photographed from multiple viewpoints and at different timings. For each timing, a 360 degrees environment map is captured together with a colour-calibration chequerboard to allow accurate numerical evaluations on real data against ground truth. Comparisons against state of the art show that NeRF-OSR enables controllable lighting and viewpoint editing at higher quality and with realistic self-shadowing reproduction. Our method and the dataset will be made publicly available at https://4dqv.mpi-inf.mpg.de/NeRF-OSR/.

Related papers

• EverLight: Indoor-Outdoor Editable HDR Lighting Estimation [9.443561684223514]
  We propose a method which combines a parametric light model with 360deg panoramas, ready to use as HDRI in rendering engines. In our representation, users can easily edit light direction, intensity, number, etc. to impact shading while providing rich, complex reflections while seamlessly blending with the edits.
  arXiv  Detail & Related papers  (2023-04-26T00:20:59Z)
• Neural Light Field Estimation for Street Scenes with Differentiable Virtual Object Insertion [129.52943959497665]
  Existing works on outdoor lighting estimation typically simplify the scene lighting into an environment map. We propose a neural approach that estimates the 5D HDR light field from a single image. We show the benefits of our AR object insertion in an autonomous driving application.
  arXiv  Detail & Related papers  (2022-08-19T17:59:16Z)
• Factorized and Controllable Neural Re-Rendering of Outdoor Scene for Photo Extrapolation [50.00344639039158]
  We propose a factorized neural re-rendering model to produce novel views from cluttered outdoor Internet photo collections. We also propose a novel realism augmentation process to complement appearance details, which automatically propagates the texture details from a narrow captured photo to the extrapolated neural rendered image.
  arXiv  Detail & Related papers  (2022-07-14T13:28:08Z)
• Physically-Based Editing of Indoor Scene Lighting from a Single Image [106.60252793395104]
  We present a method to edit complex indoor lighting from a single image with its predicted depth and light source segmentation masks. We tackle this problem using two novel components: 1) a holistic scene reconstruction method that estimates scene reflectance and parametric 3D lighting, and 2) a neural rendering framework that re-renders the scene from our predictions.
  arXiv  Detail & Related papers  (2022-05-19T06:44:37Z)
• Multi-view Gradient Consistency for SVBRDF Estimation of Complex Scenes under Natural Illumination [6.282068591820945]
  This paper presents a process for estimating the spatially varying surface reflectance of complex scenes observed under natural illumination. An end-to-end process uses a model of the scene's geometry and several images capturing the scene's surfaces. Experiments show that our technique produces realistic results for arbitrary outdoor scenes with complex geometry.
  arXiv  Detail & Related papers  (2022-02-25T23:49:39Z)
• Self-supervised Outdoor Scene Relighting [92.20785788740407]
  We propose a self-supervised approach for relighting. Our approach is trained only on corpora of images collected from the internet without any user-supervision. Results show the ability of our technique to produce photo-realistic and physically plausible results, that generalizes to unseen scenes.
  arXiv  Detail & Related papers  (2021-07-07T09:46:19Z)
• A Dark Flash Normal Camera [6.686241050151697]
  Casual photography is often performed in uncontrolled lighting that can result in low quality images and degrade the performance of downstream processing. We consider the problem of estimating surface normal and reflectance maps of scenes depicting people despite these conditions by supplementing the available visible illumination with a single near infrared (NIR) light source and camera, a so-called "dark flash image" Our method takes as input a single color image captured under arbitrary visible lighting and a single dark flash image captured under controlled front-lit NIR lighting at the same viewpoint, and computes a normal map, a diffuse albedo map, and a specular intensity map of the
  arXiv  Detail & Related papers  (2020-12-11T05:08:22Z)
• Neural Reflectance Fields for Appearance Acquisition [61.542001266380375]
  We present Neural Reflectance Fields, a novel deep scene representation that encodes volume density, normal and reflectance properties at any 3D point in a scene. We combine this representation with a physically-based differentiable ray marching framework that can render images from a neural reflectance field under any viewpoint and light.
  arXiv  Detail & Related papers  (2020-08-09T22:04:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.