Baking Relightable NeRF for Real-time Direct/Indirect Illumination Rendering

• URL: http://arxiv.org/abs/2409.10327v1
• Date: Mon, 16 Sep 2024 14:38:26 GMT
• Title: Baking Relightable NeRF for Real-time Direct/Indirect Illumination Rendering
• Authors: Euntae Choi, Vincent Carpentier, Seunghun Shin, Sungjoo Yoo,
• Abstract summary: Real-time relighting is challenging due to high computation cost of the rendering equation. We propose a novel method that executes a CNN to compute primary surface points and rendering parameters. Both distillations are trained from a pre-trained teacher model and provide real-time physically-based rendering under unseen lighting condition.
• Score: 4.812321790984493
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Relighting, which synthesizes a novel view under a given lighting condition (unseen in training time), is a must feature for immersive photo-realistic experience. However, real-time relighting is challenging due to high computation cost of the rendering equation which requires shape and material decomposition and visibility test to model shadow. Additionally, for indirect illumination, additional computation of rendering equation on each secondary surface point (where reflection occurs) is required rendering real-time relighting challenging. We propose a novel method that executes a CNN renderer to compute primary surface points and rendering parameters, required for direct illumination. We also present a lightweight hash grid-based renderer, for indirect illumination, which is recursively executed to perform the secondary ray tracing process. Both renderers are trained in a distillation from a pre-trained teacher model and provide real-time physically-based rendering under unseen lighting condition at a negligible loss of rendering quality.

Related papers

• NeAI: A Pre-convoluted Representation for Plug-and-Play Neural Ambient Illumination [28.433403714053103]
  We propose a framework named neural ambient illumination (NeAI) NeAI uses Neural Radiance Fields (NeRF) as a lighting model to handle complex lighting in a physically based way. Experiments demonstrate the superior performance of novel-view rendering compared to previous works.
  arXiv  Detail & Related papers  (2023-04-18T06:32:30Z)
• RelightableHands: Efficient Neural Relighting of Articulated Hand Models [46.60594572471557]
  We present the first neural relighting approach for rendering high-fidelity personalized hands that can be animated in real-time under novel illumination. Our approach adopts a teacher-student framework, where the teacher learns appearance under a single point light from images captured in a light-stage. Using images rendered by the teacher model as training data, an efficient student model directly predicts appearance under natural illuminations in real-time.
  arXiv  Detail & Related papers  (2023-02-09T18:59:48Z)
• Physics-based Indirect Illumination for Inverse Rendering [70.27534648770057]
  We present a physics-based inverse rendering method that learns the illumination, geometry, and materials of a scene from posed multi-view RGB images. As a side product, our physics-based inverse rendering model also facilitates flexible and realistic material editing as well as relighting.
  arXiv  Detail & Related papers  (2022-12-09T07:33:49Z)
• Neural Radiance Transfer Fields for Relightable Novel-view Synthesis with Global Illumination [63.992213016011235]
  We propose a method for scene relighting under novel views by learning a neural precomputed radiance transfer function. Our method can be solely supervised on a set of real images of the scene under a single unknown lighting condition. Results show that the recovered disentanglement of scene parameters improves significantly over the current state of the art.
  arXiv  Detail & Related papers  (2022-07-27T16:07:48Z)
• Physically-Based Editing of Indoor Scene Lighting from a Single Image [106.60252793395104]
  We present a method to edit complex indoor lighting from a single image with its predicted depth and light source segmentation masks. We tackle this problem using two novel components: 1) a holistic scene reconstruction method that estimates scene reflectance and parametric 3D lighting, and 2) a neural rendering framework that re-renders the scene from our predictions.
  arXiv  Detail & Related papers  (2022-05-19T06:44:37Z)
• Modeling Indirect Illumination for Inverse Rendering [31.734819333921642]
  In this paper, we propose a novel approach to efficiently recovering spatially-varying indirect illumination. The key insight is that indirect illumination can be conveniently derived from the neural radiance field learned from input images. Experiments on both synthetic and real data demonstrate the superior performance of our approach compared to previous work.
  arXiv  Detail & Related papers  (2022-04-14T09:10:55Z)
• DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderer [78.91753256634453]
  We consider the challenging problem of predicting intrinsic object properties from a single image by exploiting differentiables. In this work, we propose DIBR++, a hybrid differentiable which supports these effects by combining specularization and ray-tracing. Compared to more advanced physics-based differentiables, DIBR++ is highly performant due to its compact and expressive model.
  arXiv  Detail & Related papers  (2021-10-30T01:59:39Z)
• Learning Neural Transmittance for Efficient Rendering of Reflectance Fields [43.24427791156121]
  We propose a novel method based on precomputed Neural Transmittance Functions to accelerate rendering of neural reflectance fields. Results on real and synthetic scenes demonstrate almost two order of magnitude speedup for renderings under environment maps with minimal accuracy loss.
  arXiv  Detail & Related papers  (2021-10-25T21:12:25Z)
• Sparse Needlets for Lighting Estimation with Spherical Transport Loss [89.52531416604774]
  NeedleLight is a new lighting estimation model that represents illumination with needlets and allows lighting estimation in both frequency domain and spatial domain jointly. Extensive experiments show that NeedleLight achieves superior lighting estimation consistently across multiple evaluation metrics as compared with state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-06-24T15:19:42Z)
• Object-based Illumination Estimation with Rendering-aware Neural Networks [56.01734918693844]
  We present a scheme for fast environment light estimation from the RGBD appearance of individual objects and their local image areas. With the estimated lighting, virtual objects can be rendered in AR scenarios with shading that is consistent to the real scene.
  arXiv  Detail & Related papers  (2020-08-06T08:23:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.