IRIS: Inverse Rendering of Indoor Scenes from Low Dynamic Range Images

• URL: http://arxiv.org/abs/2401.12977v1
• Date: Tue, 23 Jan 2024 18:59:56 GMT
• Title: IRIS: Inverse Rendering of Indoor Scenes from Low Dynamic Range Images
• Authors: Zhi-Hao Lin, Jia-Bin Huang, Zhengqin Li, Zhao Dong, Christian Richardt, Tuotuo Li, Michael Zollh\"ofer, Johannes Kopf, Shenlong Wang, Changil Kim
• Abstract summary: We present a method that recovers the physically based material properties and lighting of a scene from multi-view, low-dynamic-range (LDR) images. Our method outperforms existing methods taking LDR images as input, and allows for highly realistic relighting and object insertion.
• Score: 32.83096814910201
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While numerous 3D reconstruction and novel-view synthesis methods allow for photorealistic rendering of a scene from multi-view images easily captured with consumer cameras, they bake illumination in their representations and fall short of supporting advanced applications like material editing, relighting, and virtual object insertion. The reconstruction of physically based material properties and lighting via inverse rendering promises to enable such applications. However, most inverse rendering techniques require high dynamic range (HDR) images as input, a setting that is inaccessible to most users. We present a method that recovers the physically based material properties and spatially-varying HDR lighting of a scene from multi-view, low-dynamic-range (LDR) images. We model the LDR image formation process in our inverse rendering pipeline and propose a novel optimization strategy for material, lighting, and a camera response model. We evaluate our approach with synthetic and real scenes compared to the state-of-the-art inverse rendering methods that take either LDR or HDR input. Our method outperforms existing methods taking LDR images as input, and allows for highly realistic relighting and object insertion.

Related papers

• Relighting Scenes with Object Insertions in Neural Radiance Fields [24.18050535794117]
  We propose a novel NeRF-based pipeline for inserting object NeRFs into scene NeRFs. The proposed method achieves realistic relighting effects in extensive experimental evaluations.
  arXiv  Detail & Related papers  (2024-06-21T00:58:58Z)
• Cinematic Gaussians: Real-Time HDR Radiance Fields with Depth of Field [23.92087253022495]
  Radiance field methods represent the state of the art in reconstructing complex scenes from multi-view photos. Their reliance on a pinhole camera model, assuming all scene elements are in focus in the input images, presents practical challenges and complicates refocusing during novel-view synthesis. We present a lightweight analytical based on 3D Gaussian Splatting that utilizes multi-view LDR images on varying exposure times, radiance of apertures, and focus distances as input to reconstruct a high-dynamic-range scene.
  arXiv  Detail & Related papers  (2024-06-11T15:00:24Z)
• Pano-NeRF: Synthesizing High Dynamic Range Novel Views with Geometry from Sparse Low Dynamic Range Panoramic Images [82.1477261107279]
  We propose the irradiance fields from sparse LDR panoramic images to increase the observation counts for faithful geometry recovery. Experiments demonstrate that the irradiance fields outperform state-of-the-art methods on both geometry recovery and HDR reconstruction.
  arXiv  Detail & Related papers  (2023-12-26T08:10:22Z)
• Spatiotemporally Consistent HDR Indoor Lighting Estimation [66.26786775252592]
  We propose a physically-motivated deep learning framework to solve the indoor lighting estimation problem. Given a single LDR image with a depth map, our method predicts spatially consistent lighting at any given image position. Our framework achieves photorealistic lighting prediction with higher quality compared to state-of-the-art single-image or video-based methods.
  arXiv  Detail & Related papers  (2023-05-07T20:36:29Z)
• Neural Fields meet Explicit Geometric Representation for Inverse Rendering of Urban Scenes [62.769186261245416]
  We present a novel inverse rendering framework for large urban scenes capable of jointly reconstructing the scene geometry, spatially-varying materials, and HDR lighting from a set of posed RGB images with optional depth. Specifically, we use a neural field to account for the primary rays, and use an explicit mesh (reconstructed from the underlying neural field) for modeling secondary rays that produce higher-order lighting effects such as cast shadows.
  arXiv  Detail & Related papers  (2023-04-06T17:51:54Z)
• ENVIDR: Implicit Differentiable Renderer with Neural Environment Lighting [9.145875902703345]
  We introduce ENVIDR, a rendering and modeling framework for high-quality rendering and reconstruction of surfaces with challenging specular reflections. We first propose a novel neural with decomposed rendering to learn the interaction between surface and environment lighting. We then propose an SDF-based neural surface model that leverages this learned neural to represent general scenes.
  arXiv  Detail & Related papers  (2023-03-23T04:12:07Z)
• MAIR: Multi-view Attention Inverse Rendering with 3D Spatially-Varying Lighting Estimation [13.325800282424598]
  We propose a scene-level inverse rendering framework that uses multi-view images to decompose the scene into geometry, a SVBRDF, and 3D spatially-varying lighting. Our experiments show that the proposed method achieves better performance than single-view-based methods, but also achieves robust performance on unseen real-world scene.
  arXiv  Detail & Related papers  (2023-03-22T08:07:28Z)
• DynIBaR: Neural Dynamic Image-Based Rendering [79.44655794967741]
  We address the problem of synthesizing novel views from a monocular video depicting a complex dynamic scene. We adopt a volumetric image-based rendering framework that synthesizes new viewpoints by aggregating features from nearby views. We demonstrate significant improvements over state-of-the-art methods on dynamic scene datasets.
  arXiv  Detail & Related papers  (2022-11-20T20:57:02Z)
• PhotoScene: Photorealistic Material and Lighting Transfer for Indoor Scenes [84.66946637534089]
  PhotoScene is a framework that takes input image(s) of a scene and builds a photorealistic digital twin with high-quality materials and similar lighting. We model scene materials using procedural material graphs; such graphs represent photorealistic and resolution-independent materials. We evaluate our technique on objects and layout reconstructions from ScanNet, SUN RGB-D and stock photographs, and demonstrate that our method reconstructs high-quality, fully relightable 3D scenes.
  arXiv  Detail & Related papers  (2022-07-02T06:52:44Z)
• Neural Reflectance Fields for Appearance Acquisition [61.542001266380375]
  We present Neural Reflectance Fields, a novel deep scene representation that encodes volume density, normal and reflectance properties at any 3D point in a scene. We combine this representation with a physically-based differentiable ray marching framework that can render images from a neural reflectance field under any viewpoint and light.
  arXiv  Detail & Related papers  (2020-08-09T22:04:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.