Ref-NeuS: Ambiguity-Reduced Neural Implicit Surface Learning for Multi-View Reconstruction with Reflection

• URL: http://arxiv.org/abs/2303.10840v2
• Date: Mon, 17 Jul 2023 07:24:30 GMT
• Title: Ref-NeuS: Ambiguity-Reduced Neural Implicit Surface Learning for Multi-View Reconstruction with Reflection
• Authors: Wenhang Ge and Tao Hu and Haoyu Zhao and Shu Liu and Ying-Cong Chen
• Abstract summary: Ref-NeuS aims to reduce ambiguity by attenuating the effect of reflective surfaces. We show that our model achieves high-quality surface reconstruction on reflective surfaces and outperforms the state-of-the-arts by a large margin.
• Score: 24.23826907954389
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Neural implicit surface learning has shown significant progress in multi-view 3D reconstruction, where an object is represented by multilayer perceptrons that provide continuous implicit surface representation and view-dependent radiance. However, current methods often fail to accurately reconstruct reflective surfaces, leading to severe ambiguity. To overcome this issue, we propose Ref-NeuS, which aims to reduce ambiguity by attenuating the effect of reflective surfaces. Specifically, we utilize an anomaly detector to estimate an explicit reflection score with the guidance of multi-view context to localize reflective surfaces. Afterward, we design a reflection-aware photometric loss that adaptively reduces ambiguity by modeling rendered color as a Gaussian distribution, with the reflection score representing the variance. We show that together with a reflection direction-dependent radiance, our model achieves high-quality surface reconstruction on reflective surfaces and outperforms the state-of-the-arts by a large margin. Besides, our model is also comparable on general surfaces.

Related papers

• Planar Reflection-Aware Neural Radiance Fields [32.709468082010126]
  We introduce a reflection-aware NeRF that jointly models planar reflectors, such as windows, and explicitly casts reflected rays to capture the source of the high-frequency reflections. Rendering along the primary ray results in a clean, reflection-free view, while explicitly rendering along the reflected ray allows us to reconstruct highly detailed reflections.
  arXiv  Detail & Related papers  (2024-11-07T18:55:08Z)
• Photometric Inverse Rendering: Shading Cues Modeling and Surface Reflectance Regularization [46.146783750386994]
  We propose a new method for neural inverse rendering. Our method jointly optimize the light source position to account for the self-shadows in images. To enhance surface reflectance decomposition, we introduce a new regularization.
  arXiv  Detail & Related papers  (2024-08-13T11:39:14Z)
• NeRSP: Neural 3D Reconstruction for Reflective Objects with Sparse Polarized Images [62.752710734332894]
  NeRSP is a Neural 3D reconstruction technique for Reflective surfaces with Sparse Polarized images. We derive photometric and geometric cues from the polarimetric image formation model and multiview azimuth consistency. We achieve the state-of-the-art surface reconstruction results with only 6 views as input.
  arXiv  Detail & Related papers  (2024-06-11T09:53:18Z)
• REFRAME: Reflective Surface Real-Time Rendering for Mobile Devices [51.983541908241726]
  This work tackles the challenging task of achieving real-time novel view synthesis for reflective surfaces across various scenes. Existing real-time rendering methods, especially those based on meshes, often have subpar performance in modeling surfaces with rich view-dependent appearances. We decompose the color into diffuse and specular, and model the specular color in the reflected direction based on a neural environment map.
  arXiv  Detail & Related papers  (2024-03-25T07:07:50Z)
• UniSDF: Unifying Neural Representations for High-Fidelity 3D Reconstruction of Complex Scenes with Reflections [92.38975002642455]
  We propose UniSDF, a general purpose 3D reconstruction method that can reconstruct large complex scenes with reflections. Our method is able to robustly reconstruct complex large-scale scenes with fine details and reflective surfaces.
  arXiv  Detail & Related papers  (2023-12-20T18:59:42Z)
• Diffusion Reflectance Map: Single-Image Stochastic Inverse Rendering of Illumination and Reflectance [19.20790327389337]
  Reflectance bounds the frequency spectrum of illumination in the object appearance. We introduce the first inverse rendering method, which recovers the attenuated frequency spectrum of an illumination jointly with the reflectance of an object of known geometry.
  arXiv  Detail & Related papers  (2023-12-07T18:50:00Z)
• Predicting Surface Reflectance Properties of Outdoor Scenes Under Unknown Natural Illumination [6.767885381740952]
  This paper proposes a complete framework to predict surface reflectance properties of outdoor scenes under unknown natural illumination. We recast the problem into its two constituent components involving the BRDF incoming light and outgoing view directions. We present experiments that show that rendering with the predicted reflectance properties results in a visually similar appearance to using textures.
  arXiv  Detail & Related papers  (2021-05-14T13:31:47Z)
• Location-aware Single Image Reflection Removal [54.93808224890273]
  This paper proposes a novel location-aware deep learning-based single image reflection removal method. We use a reflection confidence map as the cues for the network to learn how to encode the reflection information adaptively. The integration of location information into the network significantly improves the quality of reflection removal results.
  arXiv  Detail & Related papers  (2020-12-13T19:34:35Z)
• Two-Stage Single Image Reflection Removal with Reflection-Aware Guidance [78.34235841168031]
  We present a novel two-stage network with reflection-aware guidance (RAGNet) for single image reflection removal (SIRR) RAG can be used (i) to mitigate the effect of reflection from the observation, and (ii) to generate mask in partial convolution for mitigating the effect of deviating from linear combination hypothesis. Experiments on five commonly used datasets demonstrate the quantitative and qualitative superiority of our RAGNet in comparison to the state-of-the-art SIRR methods.
  arXiv  Detail & Related papers  (2020-12-02T03:14:57Z)
• Polarized Reflection Removal with Perfect Alignment in the Wild [66.48211204364142]
  We present a novel formulation to removing reflection from polarized images in the wild. We first identify the misalignment issues of existing reflection removal datasets. We build a new dataset with more than 100 types of glass in which obtained transmission images are perfectly aligned with input mixed images.
  arXiv  Detail & Related papers  (2020-03-28T13:29:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.