ENVIDR: Implicit Differentiable Renderer with Neural Environment Lighting

• URL: http://arxiv.org/abs/2303.13022v1
• Date: Thu, 23 Mar 2023 04:12:07 GMT
• Title: ENVIDR: Implicit Differentiable Renderer with Neural Environment Lighting
• Authors: Ruofan Liang, Huiting Chen, Chunlin Li, Fan Chen, Selvakumar Panneer, Nandita Vijaykumar
• Abstract summary: We introduce ENVIDR, a rendering and modeling framework for high-quality rendering and reconstruction of surfaces with challenging specular reflections. We first propose a novel neural with decomposed rendering to learn the interaction between surface and environment lighting. We then propose an SDF-based neural surface model that leverages this learned neural to represent general scenes.
• Score: 9.145875902703345
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Recent advances in neural rendering have shown great potential for reconstructing scenes from multiview images. However, accurately representing objects with glossy surfaces remains a challenge for existing methods. In this work, we introduce ENVIDR, a rendering and modeling framework for high-quality rendering and reconstruction of surfaces with challenging specular reflections. To achieve this, we first propose a novel neural renderer with decomposed rendering components to learn the interaction between surface and environment lighting. This renderer is trained using existing physically based renderers and is decoupled from actual scene representations. We then propose an SDF-based neural surface model that leverages this learned neural renderer to represent general scenes. Our model additionally synthesizes indirect illuminations caused by inter-reflections from shiny surfaces by marching surface-reflected rays. We demonstrate that our method outperforms state-of-art methods on challenging shiny scenes, providing high-quality rendering of specular reflections while also enabling material editing and scene relighting.

Related papers

• Relighting Scenes with Object Insertions in Neural Radiance Fields [24.18050535794117]
  We propose a novel NeRF-based pipeline for inserting object NeRFs into scene NeRFs. The proposed method achieves realistic relighting effects in extensive experimental evaluations.
  arXiv  Detail & Related papers  (2024-06-21T00:58:58Z)
• NeRF-Casting: Improved View-Dependent Appearance with Consistent Reflections [57.63028964831785]
  Recent works have improved NeRF's ability to render detailed specular appearance of distant environment illumination, but are unable to synthesize consistent reflections of closer content. We address these issues with an approach based on ray tracing. Instead of querying an expensive neural network for the outgoing view-dependent radiance at points along each camera ray, our model casts rays from these points and traces them through the NeRF representation to render feature vectors.
  arXiv  Detail & Related papers  (2024-05-23T17:59:57Z)
• REFRAME: Reflective Surface Real-Time Rendering for Mobile Devices [51.983541908241726]
  This work tackles the challenging task of achieving real-time novel view synthesis for reflective surfaces across various scenes. Existing real-time rendering methods, especially those based on meshes, often have subpar performance in modeling surfaces with rich view-dependent appearances. We decompose the color into diffuse and specular, and model the specular color in the reflected direction based on a neural environment map.
  arXiv  Detail & Related papers  (2024-03-25T07:07:50Z)
• Neural Fields meet Explicit Geometric Representation for Inverse Rendering of Urban Scenes [62.769186261245416]
  We present a novel inverse rendering framework for large urban scenes capable of jointly reconstructing the scene geometry, spatially-varying materials, and HDR lighting from a set of posed RGB images with optional depth. Specifically, we use a neural field to account for the primary rays, and use an explicit mesh (reconstructed from the underlying neural field) for modeling secondary rays that produce higher-order lighting effects such as cast shadows.
  arXiv  Detail & Related papers  (2023-04-06T17:51:54Z)
• DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderer [78.91753256634453]
  We consider the challenging problem of predicting intrinsic object properties from a single image by exploiting differentiables. In this work, we propose DIBR++, a hybrid differentiable which supports these effects by combining specularization and ray-tracing. Compared to more advanced physics-based differentiables, DIBR++ is highly performant due to its compact and expressive model.
  arXiv  Detail & Related papers  (2021-10-30T01:59:39Z)
• NeRS: Neural Reflectance Surfaces for Sparse-view 3D Reconstruction in the Wild [80.09093712055682]
  We introduce a surface analog of implicit models called Neural Reflectance Surfaces (NeRS) NeRS learns a neural shape representation of a closed surface that is diffeomorphic to a sphere, guaranteeing water-tight reconstructions. We demonstrate that surface-based neural reconstructions enable learning from such data, outperforming volumetric neural rendering-based reconstructions.
  arXiv  Detail & Related papers  (2021-10-14T17:59:58Z)
• Neural Ray-Tracing: Learning Surfaces and Reflectance for Relighting and View Synthesis [28.356700318603565]
  We explicitly model the light transport between scene surfaces and we rely on traditional integration schemes and the rendering equation to reconstruct a scene. By learning decomposed transport with surface representations established in conventional rendering methods, the method naturally facilitates editing shape, reflectance, lighting and scene composition. We validate the proposed approach for scene editing, relighting and reflectance estimation learned from synthetic and captured views on a subset of NeRV's datasets.
  arXiv  Detail & Related papers  (2021-04-28T03:47:48Z)
• Neural Lumigraph Rendering [33.676795978166375]
  State-of-the-art (SOTA) neural volume rendering approaches are slow to train and require minutes of inference (i.e., rendering) time for high image resolutions. We adopt high-capacity neural scene representations with periodic activations for jointly optimizing an implicit surface and a radiance field of a scene supervised exclusively with posed 2D images. Our neural rendering pipeline accelerates SOTA neural volume rendering by about two orders of magnitude and our implicit surface representation is unique in allowing us to export a mesh with view-dependent texture information.
  arXiv  Detail & Related papers  (2021-03-22T03:46:05Z)
• Neural Reflectance Fields for Appearance Acquisition [61.542001266380375]
  We present Neural Reflectance Fields, a novel deep scene representation that encodes volume density, normal and reflectance properties at any 3D point in a scene. We combine this representation with a physically-based differentiable ray marching framework that can render images from a neural reflectance field under any viewpoint and light.
  arXiv  Detail & Related papers  (2020-08-09T22:04:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.