NeRO: Neural Geometry and BRDF Reconstruction of Reflective Objects from Multiview Images

• URL: http://arxiv.org/abs/2305.17398v1
• Date: Sat, 27 May 2023 07:40:07 GMT
• Title: NeRO: Neural Geometry and BRDF Reconstruction of Reflective Objects from Multiview Images
• Authors: Yuan Liu and Peng Wang and Cheng Lin and Xiaoxiao Long and Jiepeng Wang and Lingjie Liu and Taku Komura and Wenping Wang
• Abstract summary: We present a neural rendering-based method called NeRO for reconstructing the geometry and the BRDF of reflective objects from multiview images captured in an unknown environment.
• Score: 44.1333444097976
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: We present a neural rendering-based method called NeRO for reconstructing the geometry and the BRDF of reflective objects from multiview images captured in an unknown environment. Multiview reconstruction of reflective objects is extremely challenging because specular reflections are view-dependent and thus violate the multiview consistency, which is the cornerstone for most multiview reconstruction methods. Recent neural rendering techniques can model the interaction between environment lights and the object surfaces to fit the view-dependent reflections, thus making it possible to reconstruct reflective objects from multiview images. However, accurately modeling environment lights in the neural rendering is intractable, especially when the geometry is unknown. Most existing neural rendering methods, which can model environment lights, only consider direct lights and rely on object masks to reconstruct objects with weak specular reflections. Therefore, these methods fail to reconstruct reflective objects, especially when the object mask is not available and the object is illuminated by indirect lights. We propose a two-step approach to tackle this problem. First, by applying the split-sum approximation and the integrated directional encoding to approximate the shading effects of both direct and indirect lights, we are able to accurately reconstruct the geometry of reflective objects without any object masks. Then, with the object geometry fixed, we use more accurate sampling to recover the environment lights and the BRDF of the object. Extensive experiments demonstrate that our method is capable of accurately reconstructing the geometry and the BRDF of reflective objects from only posed RGB images without knowing the environment lights and the object masks. Codes and datasets are available at https://github.com/liuyuan-pal/NeRO.

Related papers

• Multi-times Monte Carlo Rendering for Inter-reflection Reconstruction [51.911195773164245]
  Inverse rendering methods have achieved remarkable performance in reconstructing high-fidelity 3D objects with disentangled geometries, materials, and environmental light. We propose Ref-MC2 that introduces the multi-time Monte Carlo sampling which comprehensively computes the environmental illumination. We also show downstream applications, e.g., relighting and material editing, to illustrate the disentanglement ability of our method.
  arXiv  Detail & Related papers  (2024-07-08T09:27:34Z)
• Sparse multi-view hand-object reconstruction for unseen environments [31.604141859402187]
  We train our model on a synthetic hand-object dataset and evaluate directly on a real world recorded hand-object dataset with unseen objects. We show that while reconstruction of unseen hands and objects from RGB is challenging, additional views can help improve the reconstruction quality.
  arXiv  Detail & Related papers  (2024-05-02T15:01:25Z)
• UniSDF: Unifying Neural Representations for High-Fidelity 3D Reconstruction of Complex Scenes with Reflections [92.38975002642455]
  We propose UniSDF, a general purpose 3D reconstruction method that can reconstruct large complex scenes with reflections. Our method is able to robustly reconstruct complex large-scale scenes with fine details and reflective surfaces.
  arXiv  Detail & Related papers  (2023-12-20T18:59:42Z)
• DeepShaRM: Multi-View Shape and Reflectance Map Recovery Under Unknown Lighting [35.18426818323455]
  We derive a novel multi-view method, DeepShaRM, that achieves state-of-the-art accuracy on this challenging task. We introduce a novel deep reflectance map estimation network that recovers the camera-view reflectance maps. A deep shape-from-shading network then updates the geometry estimate expressed with a signed distance function.
  arXiv  Detail & Related papers  (2023-10-26T17:50:10Z)
• NeRFactor: Neural Factorization of Shape and Reflectance Under an Unknown Illumination [60.89737319987051]
  We address the problem of recovering shape and spatially-varying reflectance of an object from posed multi-view images of the object illuminated by one unknown lighting condition. This enables the rendering of novel views of the object under arbitrary environment lighting and editing of the object's material properties.
  arXiv  Detail & Related papers  (2021-06-03T16:18:01Z)
• Multi-view 3D Reconstruction of a Texture-less Smooth Surface of Unknown Generic Reflectance [86.05191217004415]
  Multi-view reconstruction of texture-less objects with unknown surface reflectance is a challenging task. This paper proposes a simple and robust solution to this problem based on a co-light scanner.
  arXiv  Detail & Related papers  (2021-05-25T01:28:54Z)
• Shape From Tracing: Towards Reconstructing 3D Object Geometry and SVBRDF Material from Images via Differentiable Path Tracing [16.975014467319443]
  Differentiable path tracing is an appealing framework as it can reproduce complex appearance effects. We show how to use differentiable ray tracing to refine an initial coarse mesh and per-mesh-facet material representation. We also show how to refine initial reconstructions of real-world objects in unconstrained environments.
  arXiv  Detail & Related papers  (2020-12-06T18:55:35Z)
• Deep 3D Capture: Geometry and Reflectance from Sparse Multi-View Images [59.906948203578544]
  We introduce a novel learning-based method to reconstruct the high-quality geometry and complex, spatially-varying BRDF of an arbitrary object. We first estimate per-view depth maps using a deep multi-view stereo network. These depth maps are used to coarsely align the different views. We propose a novel multi-view reflectance estimation network architecture.
  arXiv  Detail & Related papers  (2020-03-27T21:28:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.