REF$^2$-NeRF: Reflection and Refraction aware Neural Radiance Field
- URL: http://arxiv.org/abs/2311.17116v4
- Date: Thu, 18 Apr 2024 13:03:44 GMT
- Title: REF$^2$-NeRF: Reflection and Refraction aware Neural Radiance Field
- Authors: Wooseok Kim, Taiki Fukiage, Takeshi Oishi,
- Abstract summary: This paper proposes a NeRF-based modeling method for scenes containing a glass case.
Re refraction and reflection are modeled using elements that are dependent and independent of the viewer's perspective.
Compared to existing methods, the proposed method enables more accurate modeling of both glass refraction and the overall scene.
- Score: 4.714335699701277
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Recently, significant progress has been made in the study of methods for 3D reconstruction from multiple images using implicit neural representations, exemplified by the neural radiance field (NeRF) method. Such methods, which are based on volume rendering, can model various light phenomena, and various extended methods have been proposed to accommodate different scenes and situations. However, when handling scenes with multiple glass objects, e.g., objects in a glass showcase, modeling the target scene accurately has been challenging due to the presence of multiple reflection and refraction effects. Thus, this paper proposes a NeRF-based modeling method for scenes containing a glass case. In the proposed method, refraction and reflection are modeled using elements that are dependent and independent of the viewer's perspective. This approach allows us to estimate the surfaces where refraction occurs, i.e., glass surfaces, and enables the separation and modeling of both direct and reflected light components. The proposed method requires predetermined camera poses, but accurately estimating these poses in scenes with glass objects is difficult. Therefore, we used a robotic arm with an attached camera to acquire images with known poses. Compared to existing methods, the proposed method enables more accurate modeling of both glass refraction and the overall scene.
Related papers
- Multi-times Monte Carlo Rendering for Inter-reflection Reconstruction [51.911195773164245]
Inverse rendering methods have achieved remarkable performance in reconstructing high-fidelity 3D objects with disentangled geometries, materials, and environmental light.
We propose Ref-MC2 that introduces the multi-time Monte Carlo sampling which comprehensively computes the environmental illumination.
We also show downstream applications, e.g., relighting and material editing, to illustrate the disentanglement ability of our method.
arXiv Detail & Related papers (2024-07-08T09:27:34Z) - Neural Gaffer: Relighting Any Object via Diffusion [43.87941408722868]
We propose a novel end-to-end 2D relighting diffusion model, called Neural Gaffer.
Our model takes a single image of any object and can synthesize an accurate, high-quality relit image under any novel lighting condition.
We evaluate our model on both synthetic and in-the-wild Internet imagery and demonstrate its advantages in terms of generalization and accuracy.
arXiv Detail & Related papers (2024-06-11T17:50:15Z) - Seeing Through the Glass: Neural 3D Reconstruction of Object Inside a
Transparent Container [61.50401406132946]
Transparent enclosures pose challenges of multiple light reflections and refractions at the interface between different propagation media.
We use an existing neural reconstruction method (NeuS) that implicitly represents the geometry and appearance of the inner subspace.
In order to account for complex light interactions, we develop a hybrid rendering strategy that combines volume rendering with ray tracing.
arXiv Detail & Related papers (2023-03-24T04:58:27Z) - Self-calibrating Photometric Stereo by Neural Inverse Rendering [88.67603644930466]
This paper tackles the task of uncalibrated photometric stereo for 3D object reconstruction.
We propose a new method that jointly optimize object shape, light directions, and light intensities.
Our method demonstrates state-of-the-art accuracy in light estimation and shape recovery on real-world datasets.
arXiv Detail & Related papers (2022-07-16T02:46:15Z) - NeRFReN: Neural Radiance Fields with Reflections [16.28256369376256]
We introduce NeRFReN, which is built upon NeRF to model scenes with reflections.
We propose to split a scene into transmitted and reflected components, and model the two components with separate neural radiance fields.
Experiments on various self-captured scenes show that our method achieves high-quality novel view synthesis and physically sound depth estimation results.
arXiv Detail & Related papers (2021-11-30T09:36:00Z) - NeRFactor: Neural Factorization of Shape and Reflectance Under an
Unknown Illumination [60.89737319987051]
We address the problem of recovering shape and spatially-varying reflectance of an object from posed multi-view images of the object illuminated by one unknown lighting condition.
This enables the rendering of novel views of the object under arbitrary environment lighting and editing of the object's material properties.
arXiv Detail & Related papers (2021-06-03T16:18:01Z) - Neural Articulated Radiance Field [90.91714894044253]
We present Neural Articulated Radiance Field (NARF), a novel deformable 3D representation for articulated objects learned from images.
Experiments show that the proposed method is efficient and can generalize well to novel poses.
arXiv Detail & Related papers (2021-04-07T13:23:14Z) - Refractive Light-Field Features for Curved Transparent Objects in
Structure from Motion [10.380414189465345]
We propose a novel image feature for light fields that detects and describes the patterns of light refracted through curved transparent objects.
We demonstrate improved structure-from-motion performance in challenging scenes containing refractive objects.
Our method is a critical step towards allowing robots to operate around refractive objects.
arXiv Detail & Related papers (2021-03-29T05:55:32Z) - Monocular Reconstruction of Neural Face Reflectance Fields [0.0]
The reflectance field of a face describes the reflectance properties responsible for complex lighting effects.
Most existing methods for estimating the face reflectance from a monocular image assume faces to be diffuse with very few approaches adding a specular component.
We present a new neural representation for face reflectance where we can estimate all components of the reflectance responsible for the final appearance from a single monocular image.
arXiv Detail & Related papers (2020-08-24T08:19:05Z) - Polarized Reflection Removal with Perfect Alignment in the Wild [66.48211204364142]
We present a novel formulation to removing reflection from polarized images in the wild.
We first identify the misalignment issues of existing reflection removal datasets.
We build a new dataset with more than 100 types of glass in which obtained transmission images are perfectly aligned with input mixed images.
arXiv Detail & Related papers (2020-03-28T13:29:31Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.