NeMI: Unifying Neural Radiance Fields with Multiplane Images for Novel View Synthesis

• URL: http://arxiv.org/abs/2103.14910v1
• Date: Sat, 27 Mar 2021 13:41:00 GMT
• Title: NeMI: Unifying Neural Radiance Fields with Multiplane Images for Novel View Synthesis
• Authors: Jiaxin Li, Zijian Feng, Qi She, Henghui Ding, Changhu Wang, Gim Hee Lee
• Abstract summary: We propose an approach to perform novel view synthesis and depth estimation via dense 3D reconstruction from a single image. Our NeMI unifies Neural radiance fields (NeRF) with Multiplane Images (MPI) We also achieve competitive results in depth estimation on iBims-1 and NYU-v2 without annotated depth supervision.
• Score: 69.19261797333635
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this paper, we propose an approach to perform novel view synthesis and depth estimation via dense 3D reconstruction from a single image. Our NeMI unifies Neural radiance fields (NeRF) with Multiplane Images (MPI). Specifically, our NeMI is a general two-dimensional and image-conditioned extension of NeRF, and a continuous depth generalization of MPI. Given a single image as input, our method predicts a 4-channel image (RGB and volume density) at arbitrary depth values to jointly reconstruct the camera frustum and fill in occluded contents. The reconstructed and inpainted frustum can then be easily rendered into novel RGB or depth views using differentiable rendering. Extensive experiments on RealEstate10K, KITTI and Flowers Light Fields show that our NeMI outperforms state-of-the-art by a large margin in novel view synthesis. We also achieve competitive results in depth estimation on iBims-1 and NYU-v2 without annotated depth supervision. Project page available at https://vincentfung13.github.io/projects/nemi/

Related papers

• Learning Neural Implicit through Volume Rendering with Attentive Depth Fusion Priors [32.63878457242185]
  We learn neural implicit representations from multi-view RGBD images through volume rendering with an attentive depth fusion prior. Our attention mechanism works with either a one-time fused TSDF that represents a whole scene or an incrementally fused TSDF that represents a partial scene. Our evaluations on widely used benchmarks including synthetic and real-world scans show our superiority over the latest neural implicit methods.
  arXiv  Detail & Related papers  (2023-10-17T21:45:51Z)
• SCADE: NeRFs from Space Carving with Ambiguity-Aware Depth Estimates [16.344734292989504]
  SCADE is a novel technique that improves NeRF reconstruction quality on sparse, unconstrained input views. We propose a new method that learns to predict, for each view, a continuous, multimodal distribution of depth estimates. Experiments show that our approach enables higher fidelity novel view synthesis from sparse views.
  arXiv  Detail & Related papers  (2023-03-23T18:00:07Z)
• Behind the Scenes: Density Fields for Single View Reconstruction [63.40484647325238]
  Inferring meaningful geometric scene representation from a single image is a fundamental problem in computer vision. We propose to predict implicit density fields. A density field maps every location in the frustum of the input image to volumetric density. We show that our method is able to predict meaningful geometry for regions that are occluded in the input image.
  arXiv  Detail & Related papers  (2023-01-18T17:24:01Z)
• SceneRF: Self-Supervised Monocular 3D Scene Reconstruction with Radiance Fields [19.740018132105757]
  SceneRF is a self-supervised monocular scene reconstruction method using only posed image sequences for training. At inference, a single input image suffices to hallucinate novel depth views which are fused together to obtain 3D scene reconstruction.
  arXiv  Detail & Related papers  (2022-12-05T18:59:57Z)
• DINER: Depth-aware Image-based NEural Radiance fields [45.63488428831042]
  We present Depth-aware Image-based NEural Radiance fields (DINER) Given a sparse set of RGB input views, we predict depth and feature maps to guide the reconstruction of a scene representation. We propose novel techniques to incorporate depth information into feature fusion and efficient scene sampling.
  arXiv  Detail & Related papers  (2022-11-29T23:22:44Z)
• One-Shot Neural Fields for 3D Object Understanding [112.32255680399399]
  We present a unified and compact scene representation for robotics. Each object in the scene is depicted by a latent code capturing geometry and appearance. This representation can be decoded for various tasks such as novel view rendering, 3D reconstruction, and stable grasp prediction.
  arXiv  Detail & Related papers  (2022-10-21T17:33:14Z)
• NVS-MonoDepth: Improving Monocular Depth Prediction with Novel View Synthesis [74.4983052902396]
  We propose a novel training method split in three main steps to improve monocular depth estimation. Experimental results prove that our method achieves state-of-the-art or comparable performance on the KITTI and NYU-Depth-v2 datasets.
  arXiv  Detail & Related papers  (2021-12-22T12:21:08Z)
• Dense Depth Priors for Neural Radiance Fields from Sparse Input Views [37.92064060160628]
  We propose a method to synthesize novel views of whole rooms from an order of magnitude fewer images. Our method enables data-efficient novel view synthesis on challenging indoor scenes, using as few as 18 images for an entire scene.
  arXiv  Detail & Related papers  (2021-12-06T19:00:02Z)
• UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction [61.17219252031391]
  We present a novel method for reconstructing surfaces from multi-view images using Neural implicit 3D representations. Our key insight is that implicit surface models and radiance fields can be formulated in a unified way, enabling both surface and volume rendering. Our experiments demonstrate that we outperform NeRF in terms of reconstruction quality while performing on par with IDR without requiring masks.
  arXiv  Detail & Related papers  (2021-04-20T15:59:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.