3D Reconstruction with Generalizable Neural Fields using Scene Priors

• URL: http://arxiv.org/abs/2309.15164v2
• Date: Fri, 29 Sep 2023 02:46:17 GMT
• Title: 3D Reconstruction with Generalizable Neural Fields using Scene Priors
• Authors: Yang Fu, Shalini De Mello, Xueting Li, Amey Kulkarni, Jan Kautz, Xiaolong Wang, Sifei Liu
• Abstract summary: We introduce training generalizable Neural Fields incorporating scene Priors (NFPs) The NFP network maps any single-view RGB-D image into signed distance and radiance values. A complete scene can be reconstructed by merging individual frames in the volumetric space WITHOUT a fusion module.
• Score: 71.37871576124789
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: High-fidelity 3D scene reconstruction has been substantially advanced by recent progress in neural fields. However, most existing methods train a separate network from scratch for each individual scene. This is not scalable, inefficient, and unable to yield good results given limited views. While learning-based multi-view stereo methods alleviate this issue to some extent, their multi-view setting makes it less flexible to scale up and to broad applications. Instead, we introduce training generalizable Neural Fields incorporating scene Priors (NFPs). The NFP network maps any single-view RGB-D image into signed distance and radiance values. A complete scene can be reconstructed by merging individual frames in the volumetric space WITHOUT a fusion module, which provides better flexibility. The scene priors can be trained on large-scale datasets, allowing for fast adaptation to the reconstruction of a new scene with fewer views. NFP not only demonstrates SOTA scene reconstruction performance and efficiency, but it also supports single-image novel-view synthesis, which is underexplored in neural fields. More qualitative results are available at: https://oasisyang.github.io/neural-prior

Related papers

• Single-view Neural Radiance Fields with Depth Teacher [10.207824869802314]
  We develop a new NeRF model for novel view synthesis using only a single image as input. We propose to combine the (coarse) planar rendering and the (fine) volume rendering to achieve higher rendering quality and better generalizations.
  arXiv  Detail & Related papers  (2023-03-17T13:19:25Z)
• SceneRF: Self-Supervised Monocular 3D Scene Reconstruction with Radiance Fields [19.740018132105757]
  SceneRF is a self-supervised monocular scene reconstruction method using only posed image sequences for training. At inference, a single input image suffices to hallucinate novel depth views which are fused together to obtain 3D scene reconstruction.
  arXiv  Detail & Related papers  (2022-12-05T18:59:57Z)
• X-NeRF: Explicit Neural Radiance Field for Multi-Scene 360$^{\circ} $ Insufficient RGB-D Views [49.55319833743988]
  This paper focuses on a rarely discussed but important setting: can we train one model that can represent multiple scenes? We refer insufficient views to few extremely sparse and almost non-overlapping views. X-NeRF, a fully explicit approach which learns a general scene completion process instead of a coordinate-based mapping, is proposed.
  arXiv  Detail & Related papers  (2022-10-11T04:29:26Z)
• Vision Transformer for NeRF-Based View Synthesis from a Single Input Image [49.956005709863355]
  We propose to leverage both the global and local features to form an expressive 3D representation. To synthesize a novel view, we train a multilayer perceptron (MLP) network conditioned on the learned 3D representation to perform volume rendering. Our method can render novel views from only a single input image and generalize across multiple object categories using a single model.
  arXiv  Detail & Related papers  (2022-07-12T17:52:04Z)
• NeuralMVS: Bridging Multi-View Stereo and Novel View Synthesis [28.83180559337126]
  We propose a novel network that can recover 3D scene geometry as a distance function, together with high-resolution color images. Our method uses only a sparse set of images as input and can generalize well to novel scenes.
  arXiv  Detail & Related papers  (2021-08-09T08:59:24Z)
• Neural Rays for Occlusion-aware Image-based Rendering [108.34004858785896]
  We present a new neural representation, called Neural Ray (NeuRay), for the novel view synthesis (NVS) task with multi-view images as input. NeuRay can quickly generate high-quality novel view rendering images of unseen scenes with little finetuning.
  arXiv  Detail & Related papers  (2021-07-28T15:09:40Z)
• Stereo Radiance Fields (SRF): Learning View Synthesis for Sparse Views of Novel Scenes [48.0304999503795]
  We introduce Stereo Radiance Fields (SRF), a neural view synthesis approach that is trained end-to-end. SRF generalizes to new scenes, and requires only sparse views at test time. Experiments show that SRF learns structure instead of overfitting on a scene.
  arXiv  Detail & Related papers  (2021-04-14T15:38:57Z)
• MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo [52.329580781898116]
  We present MVSNeRF, a novel neural rendering approach that can efficiently reconstruct neural radiance fields for view synthesis. Unlike prior works on neural radiance fields that consider per-scene optimization on densely captured images, we propose a generic deep neural network that can reconstruct radiance fields from only three nearby input views via fast network inference.
  arXiv  Detail & Related papers  (2021-03-29T13:15:23Z)
• pixelNeRF: Neural Radiance Fields from One or Few Images [20.607712035278315]
  pixelNeRF is a learning framework that predicts a continuous neural scene representation conditioned on one or few input images. We conduct experiments on ShapeNet benchmarks for single image novel view synthesis tasks with held-out objects. In all cases, pixelNeRF outperforms current state-of-the-art baselines for novel view synthesis and single image 3D reconstruction.
  arXiv  Detail & Related papers  (2020-12-03T18:59:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.