Mirror-NeRF: Learning Neural Radiance Fields for Mirrors with Whitted-Style Ray Tracing

• URL: http://arxiv.org/abs/2308.03280v1
• Date: Mon, 7 Aug 2023 03:48:07 GMT
• Title: Mirror-NeRF: Learning Neural Radiance Fields for Mirrors with Whitted-Style Ray Tracing
• Authors: Junyi Zeng, Chong Bao, Rui Chen, Zilong Dong, Guofeng Zhang, Hujun Bao, Zhaopeng Cui
• Abstract summary: We present a novel neural rendering framework, named Mirror-NeRF, which is able to learn accurate geometry and reflection of the mirror. Mirror-NeRF supports various scene manipulation applications with mirrors, such as adding new objects or mirrors into the scene and synthesizing the reflections of these new objects in mirrors.
• Score: 33.852910220413655
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recently, Neural Radiance Fields (NeRF) has exhibited significant success in novel view synthesis, surface reconstruction, etc. However, since no physical reflection is considered in its rendering pipeline, NeRF mistakes the reflection in the mirror as a separate virtual scene, leading to the inaccurate reconstruction of the mirror and multi-view inconsistent reflections in the mirror. In this paper, we present a novel neural rendering framework, named Mirror-NeRF, which is able to learn accurate geometry and reflection of the mirror and support various scene manipulation applications with mirrors, such as adding new objects or mirrors into the scene and synthesizing the reflections of these new objects in mirrors, controlling mirror roughness, etc. To achieve this goal, we propose a unified radiance field by introducing the reflection probability and tracing rays following the light transport model of Whitted Ray Tracing, and also develop several techniques to facilitate the learning process. Experiments and comparisons on both synthetic and real datasets demonstrate the superiority of our method. The code and supplementary material are available on the project webpage: https://zju3dv.github.io/Mirror-NeRF/.

Related papers

• NeRFs are Mirror Detectors: Using Structural Similarity for Multi-View Mirror Scene Reconstruction with 3D Surface Primitives [7.116175288307167]
  We present NeRF-MD, a method which shows that NeRFs can be considered as mirror detectors. We first compute an initial estimate of the scene geometry by training a standard NeRF. We then jointly optimize the radiance field and mirror geometry in a second training stage to refine their quality.
  arXiv  Detail & Related papers  (2025-01-07T18:59:53Z)
• Planar Reflection-Aware Neural Radiance Fields [32.709468082010126]
  We introduce a reflection-aware NeRF that jointly models planar reflectors, such as windows, and explicitly casts reflected rays to capture the source of the high-frequency reflections. Rendering along the primary ray results in a clean, reflection-free view, while explicitly rendering along the reflected ray allows us to reconstruct highly detailed reflections.
  arXiv  Detail & Related papers  (2024-11-07T18:55:08Z)
• Reflecting Reality: Enabling Diffusion Models to Produce Faithful Mirror Reflections [26.02117310176884]
  We tackle the problem of generating highly realistic and plausible mirror reflections using diffusion-based generative models. We propose a novel depth-conditioned inpainting method called MirrorFusion, which generates high-quality, realistic, shape and appearance-aware reflections of real-world objects. MirrorFusion outperforms state-of-the-art methods on SynMirror, as demonstrated by extensive quantitative and qualitative analysis.
  arXiv  Detail & Related papers  (2024-09-23T02:59:07Z)
• NeRF-Casting: Improved View-Dependent Appearance with Consistent Reflections [57.63028964831785]
  Recent works have improved NeRF's ability to render detailed specular appearance of distant environment illumination, but are unable to synthesize consistent reflections of closer content. We address these issues with an approach based on ray tracing. Instead of querying an expensive neural network for the outgoing view-dependent radiance at points along each camera ray, our model casts rays from these points and traces them through the NeRF representation to render feature vectors.
  arXiv  Detail & Related papers  (2024-05-23T17:59:57Z)
• MirrorGaussian: Reflecting 3D Gaussians for Reconstructing Mirror Reflections [58.003014868772254]
  MirrorGaussian is the first method for mirror scene reconstruction with real-time rendering based on 3D Gaussian Splatting. We introduce an intuitive dual-rendering strategy that enables differentiableization of both the real-world 3D Gaussians and the mirrored counterpart. Our approach significantly outperforms existing methods, achieving state-of-the-art results.
  arXiv  Detail & Related papers  (2024-05-20T09:58:03Z)
• Mirror-3DGS: Incorporating Mirror Reflections into 3D Gaussian Splatting [27.361324194709155]
  Mirror-3DGS is a novel framework designed to accurately handle mirror geometries and reflections. By incorporating mirror attributes into 3DGS, Mirror-3DGS simulates a mirrored viewpoint from behind the mirror, enhancing the realism of scene renderings.
  arXiv  Detail & Related papers  (2024-04-01T15:16:33Z)
• Revisiting Single Image Reflection Removal In the Wild [83.42368937164473]
  This research focuses on the issue of single-image reflection removal (SIRR) in real-world conditions. We devise an advanced reflection collection pipeline that is highly adaptable to a wide range of real-world reflection scenarios. We develop a large-scale, high-quality reflection dataset named Reflection Removal in the Wild (RRW)
  arXiv  Detail & Related papers  (2023-11-29T02:31:10Z)
• Multi-Space Neural Radiance Fields [74.46513422075438]
  Existing Neural Radiance Fields (NeRF) methods suffer from the existence of reflective objects. We propose a multi-space neural radiance field (MS-NeRF) that represents the scene using a group of feature fields in parallel sub-spaces. Our approach significantly outperforms the existing single-space NeRF methods for rendering high-quality scenes.
  arXiv  Detail & Related papers  (2023-05-07T13:11:07Z)
• NeRFReN: Neural Radiance Fields with Reflections [16.28256369376256]
  We introduce NeRFReN, which is built upon NeRF to model scenes with reflections. We propose to split a scene into transmitted and reflected components, and model the two components with separate neural radiance fields. Experiments on various self-captured scenes show that our method achieves high-quality novel view synthesis and physically sound depth estimation results.
  arXiv  Detail & Related papers  (2021-11-30T09:36:00Z)
• NeRS: Neural Reflectance Surfaces for Sparse-view 3D Reconstruction in the Wild [80.09093712055682]
  We introduce a surface analog of implicit models called Neural Reflectance Surfaces (NeRS) NeRS learns a neural shape representation of a closed surface that is diffeomorphic to a sphere, guaranteeing water-tight reconstructions. We demonstrate that surface-based neural reconstructions enable learning from such data, outperforming volumetric neural rendering-based reconstructions.
  arXiv  Detail & Related papers  (2021-10-14T17:59:58Z)
• NeRFactor: Neural Factorization of Shape and Reflectance Under an Unknown Illumination [60.89737319987051]
  We address the problem of recovering shape and spatially-varying reflectance of an object from posed multi-view images of the object illuminated by one unknown lighting condition. This enables the rendering of novel views of the object under arbitrary environment lighting and editing of the object's material properties.
  arXiv  Detail & Related papers  (2021-06-03T16:18:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.