NeuralMVS: Bridging Multi-View Stereo and Novel View Synthesis

• URL: http://arxiv.org/abs/2108.03880v1
• Date: Mon, 9 Aug 2021 08:59:24 GMT
• Title: NeuralMVS: Bridging Multi-View Stereo and Novel View Synthesis
• Authors: Radu Alexandru Rosu and Sven Behnke
• Abstract summary: We propose a novel network that can recover 3D scene geometry as a distance function, together with high-resolution color images. Our method uses only a sparse set of images as input and can generalize well to novel scenes.
• Score: 28.83180559337126
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multi-View Stereo (MVS) is a core task in 3D computer vision. With the surge of novel deep learning methods, learned MVS has surpassed the accuracy of classical approaches, but still relies on building a memory intensive dense cost volume. Novel View Synthesis (NVS) is a parallel line of research and has recently seen an increase in popularity with Neural Radiance Field (NeRF) models, which optimize a per scene radiance field. However, NeRF methods do not generalize to novel scenes and are slow to train and test. We propose to bridge the gap between these two methodologies with a novel network that can recover 3D scene geometry as a distance function, together with high-resolution color images. Our method uses only a sparse set of images as input and can generalize well to novel scenes. Additionally, we propose a coarse-to-fine sphere tracing approach in order to significantly increase speed. We show on various datasets that our method reaches comparable accuracy to per-scene optimized methods while being able to generalize and running significantly faster.

Related papers

• Efficient Depth-Guided Urban View Synthesis [52.841803876653465]
  We introduce Efficient Depth-Guided Urban View Synthesis (EDUS) for fast feed-forward inference and efficient per-scene fine-tuning. EDUS exploits noisy predicted geometric priors as guidance to enable generalizable urban view synthesis from sparse input images. Our results indicate that EDUS achieves state-of-the-art performance in sparse view settings when combined with fast test-time optimization.
  arXiv  Detail & Related papers  (2024-07-17T08:16:25Z)
• 3D Reconstruction with Generalizable Neural Fields using Scene Priors [71.37871576124789]
  We introduce training generalizable Neural Fields incorporating scene Priors (NFPs) The NFP network maps any single-view RGB-D image into signed distance and radiance values. A complete scene can be reconstructed by merging individual frames in the volumetric space WITHOUT a fusion module.
  arXiv  Detail & Related papers  (2023-09-26T18:01:02Z)
• Multi-Plane Neural Radiance Fields for Novel View Synthesis [5.478764356647437]
  Novel view synthesis is a long-standing problem that revolves around rendering frames of scenes from novel camera viewpoints. In this work, we examine the performance, generalization, and efficiency of single-view multi-plane neural radiance fields. We propose a new multiplane NeRF architecture that accepts multiple views to improve the synthesis results and expand the viewing range.
  arXiv  Detail & Related papers  (2023-03-03T06:32:55Z)
• SNeRF: Stylized Neural Implicit Representations for 3D Scenes [9.151746397358522]
  This paper investigates 3D scene stylization that provides a strong inductive bias for consistent novel view synthesis. We adopt the emerging neural radiance fields (NeRF) as our choice of 3D scene representation. We introduce a new training method to address this problem by alternating the NeRF and stylization optimization steps.
  arXiv  Detail & Related papers  (2022-07-05T23:45:02Z)
• Neural Deformable Voxel Grid for Fast Optimization of Dynamic View Synthesis [63.25919018001152]
  We propose a fast deformable radiance field method to handle dynamic scenes. Our method achieves comparable performance to D-NeRF using only 20 minutes for training.
  arXiv  Detail & Related papers  (2022-06-15T17:49:08Z)
• ViewFormer: NeRF-free Neural Rendering from Few Images Using Transformers [34.4824364161812]
  Novel view synthesis is a problem where we are given only a few context views sparsely covering a scene or an object. The goal is to predict novel viewpoints in the scene, which requires learning priors. We propose a 2D-only method that maps multiple context views and a query pose to a new image in a single pass of a neural network.
  arXiv  Detail & Related papers  (2022-03-18T21:08:23Z)
• Stereo Radiance Fields (SRF): Learning View Synthesis for Sparse Views of Novel Scenes [48.0304999503795]
  We introduce Stereo Radiance Fields (SRF), a neural view synthesis approach that is trained end-to-end. SRF generalizes to new scenes, and requires only sparse views at test time. Experiments show that SRF learns structure instead of overfitting on a scene.
  arXiv  Detail & Related papers  (2021-04-14T15:38:57Z)
• MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo [52.329580781898116]
  We present MVSNeRF, a novel neural rendering approach that can efficiently reconstruct neural radiance fields for view synthesis. Unlike prior works on neural radiance fields that consider per-scene optimization on densely captured images, we propose a generic deep neural network that can reconstruct radiance fields from only three nearby input views via fast network inference.
  arXiv  Detail & Related papers  (2021-03-29T13:15:23Z)
• IBRNet: Learning Multi-View Image-Based Rendering [67.15887251196894]
  We present a method that synthesizes novel views of complex scenes by interpolating a sparse set of nearby views. By drawing on source views at render time, our method hearkens back to classic work on image-based rendering.
  arXiv  Detail & Related papers  (2021-02-25T18:56:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.