S$^3$-NeRF: Neural Reflectance Field from Shading and Shadow under a Single Viewpoint

• URL: http://arxiv.org/abs/2210.08936v1
• Date: Mon, 17 Oct 2022 11:01:52 GMT
• Title: S$^3$-NeRF: Neural Reflectance Field from Shading and Shadow under a Single Viewpoint
• Authors: Wenqi Yang, Guanying Chen, Chaofeng Chen, Zhenfang Chen, Kwan-Yee K. Wong
• Abstract summary: Our method learns a neural reflectance field to represent the 3D geometry and BRDFs of a scene. Our method is capable of recovering 3D geometry, including both visible and invisible parts, of a scene from single-view images. It supports applications like novel-view synthesis and relighting.
• Score: 22.42916940712357
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we address the "dual problem" of multi-view scene reconstruction in which we utilize single-view images captured under different point lights to learn a neural scene representation. Different from existing single-view methods which can only recover a 2.5D scene representation (i.e., a normal / depth map for the visible surface), our method learns a neural reflectance field to represent the 3D geometry and BRDFs of a scene. Instead of relying on multi-view photo-consistency, our method exploits two information-rich monocular cues, namely shading and shadow, to infer scene geometry. Experiments on multiple challenging datasets show that our method is capable of recovering 3D geometry, including both visible and invisible parts, of a scene from single-view images. Thanks to the neural reflectance field representation, our method is robust to depth discontinuities. It supports applications like novel-view synthesis and relighting. Our code and model can be found at https://ywq.github.io/s3nerf.

Related papers

• One-Shot Neural Fields for 3D Object Understanding [112.32255680399399]
  We present a unified and compact scene representation for robotics. Each object in the scene is depicted by a latent code capturing geometry and appearance. This representation can be decoded for various tasks such as novel view rendering, 3D reconstruction, and stable grasp prediction.
  arXiv  Detail & Related papers  (2022-10-21T17:33:14Z)
• PS-NeRF: Neural Inverse Rendering for Multi-view Photometric Stereo [22.42916940712357]
  We present a neural inverse rendering method for MVPS based on implicit representation. Our method achieves far more accurate shape reconstruction than existing MVPS and neural rendering methods.
  arXiv  Detail & Related papers  (2022-07-23T03:55:18Z)
• Neural Groundplans: Persistent Neural Scene Representations from a Single Image [90.04272671464238]
  We present a method to map 2D image observations of a scene to a persistent 3D scene representation. We propose conditional neural groundplans as persistent and memory-efficient scene representations.
  arXiv  Detail & Related papers  (2022-07-22T17:41:24Z)
• Vision Transformer for NeRF-Based View Synthesis from a Single Input Image [49.956005709863355]
  We propose to leverage both the global and local features to form an expressive 3D representation. To synthesize a novel view, we train a multilayer perceptron (MLP) network conditioned on the learned 3D representation to perform volume rendering. Our method can render novel views from only a single input image and generalize across multiple object categories using a single model.
  arXiv  Detail & Related papers  (2022-07-12T17:52:04Z)
• Towards Learning Neural Representations from Shadows [11.60149896896201]
  We present a method that learns neural scene representations from only shadows present in the scene. Our framework is highly generalizable and can work alongside existing 3D reconstruction techniques.
  arXiv  Detail & Related papers  (2022-03-29T23:13:41Z)
• Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering [60.02806355570514]
  Inferring representations of 3D scenes from 2D observations is a fundamental problem of computer graphics, computer vision, and artificial intelligence. We propose a novel neural scene representation, Light Field Networks or LFNs, which represent both geometry and appearance of the underlying 3D scene in a 360-degree, four-dimensional light field. Rendering a ray from an LFN requires only a *single* network evaluation, as opposed to hundreds of evaluations per ray for ray-marching or based on volumetrics.
  arXiv  Detail & Related papers  (2021-06-04T17:54:49Z)
• IBRNet: Learning Multi-View Image-Based Rendering [67.15887251196894]
  We present a method that synthesizes novel views of complex scenes by interpolating a sparse set of nearby views. By drawing on source views at render time, our method hearkens back to classic work on image-based rendering.
  arXiv  Detail & Related papers  (2021-02-25T18:56:21Z)
• Single-View View Synthesis with Multiplane Images [64.46556656209769]
  We apply deep learning to generate multiplane images given two or more input images at known viewpoints. Our method learns to predict a multiplane image directly from a single image input. It additionally generates reasonable depth maps and fills in content behind the edges of foreground objects in background layers.
  arXiv  Detail & Related papers  (2020-04-23T17:59:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.