X-NeRF: Explicit Neural Radiance Field for Multi-Scene 360$^{\circ} $ Insufficient RGB-D Views

• URL: http://arxiv.org/abs/2210.05135v1
• Date: Tue, 11 Oct 2022 04:29:26 GMT
• Title: X-NeRF: Explicit Neural Radiance Field for Multi-Scene 360$^{\circ} $ Insufficient RGB-D Views
• Authors: Haoyi Zhu, Hao-Shu Fang, Cewu Lu
• Abstract summary: This paper focuses on a rarely discussed but important setting: can we train one model that can represent multiple scenes? We refer insufficient views to few extremely sparse and almost non-overlapping views. X-NeRF, a fully explicit approach which learns a general scene completion process instead of a coordinate-based mapping, is proposed.
• Score: 49.55319833743988
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Neural Radiance Fields (NeRFs), despite their outstanding performance on novel view synthesis, often need dense input views. Many papers train one model for each scene respectively and few of them explore incorporating multi-modal data into this problem. In this paper, we focus on a rarely discussed but important setting: can we train one model that can represent multiple scenes, with 360$^\circ $ insufficient views and RGB-D images? We refer insufficient views to few extremely sparse and almost non-overlapping views. To deal with it, X-NeRF, a fully explicit approach which learns a general scene completion process instead of a coordinate-based mapping, is proposed. Given a few insufficient RGB-D input views, X-NeRF first transforms them to a sparse point cloud tensor and then applies a 3D sparse generative Convolutional Neural Network (CNN) to complete it to an explicit radiance field whose volumetric rendering can be conducted fast without running networks during inference. To avoid overfitting, besides common rendering loss, we apply perceptual loss as well as view augmentation through random rotation on point clouds. The proposed methodology significantly out-performs previous implicit methods in our setting, indicating the great potential of proposed problem and approach. Codes and data are available at https://github.com/HaoyiZhu/XNeRF.

Related papers

• PERF: Panoramic Neural Radiance Field from a Single Panorama [109.31072618058043]
  PERF is a novel view synthesis framework that trains a panoramic neural radiance field from a single panorama. We propose a novel collaborative RGBD inpainting method and a progressive inpainting-and-erasing method to lift up a 360-degree 2D scene to a 3D scene. Our PERF can be widely used for real-world applications, such as panorama-to-3D, text-to-3D, and 3D scene stylization applications.
  arXiv  Detail & Related papers  (2023-10-25T17:59:01Z)
• 3D Reconstruction with Generalizable Neural Fields using Scene Priors [71.37871576124789]
  We introduce training generalizable Neural Fields incorporating scene Priors (NFPs) The NFP network maps any single-view RGB-D image into signed distance and radiance values. A complete scene can be reconstructed by merging individual frames in the volumetric space WITHOUT a fusion module.
  arXiv  Detail & Related papers  (2023-09-26T18:01:02Z)
• ViP-NeRF: Visibility Prior for Sparse Input Neural Radiance Fields [9.67057831710618]
  Training neural radiance fields (NeRFs) on sparse input views leads to overfitting and incorrect scene depth estimation. We reformulate the NeRF to also directly output the visibility of a 3D point from a given viewpoint to reduce the training time with the visibility constraint. Our model outperforms the competing sparse input NeRF models including those that use learned priors.
  arXiv  Detail & Related papers  (2023-04-28T18:26:23Z)
• SCADE: NeRFs from Space Carving with Ambiguity-Aware Depth Estimates [16.344734292989504]
  SCADE is a novel technique that improves NeRF reconstruction quality on sparse, unconstrained input views. We propose a new method that learns to predict, for each view, a continuous, multimodal distribution of depth estimates. Experiments show that our approach enables higher fidelity novel view synthesis from sparse views.
  arXiv  Detail & Related papers  (2023-03-23T18:00:07Z)
• One-Shot Neural Fields for 3D Object Understanding [112.32255680399399]
  We present a unified and compact scene representation for robotics. Each object in the scene is depicted by a latent code capturing geometry and appearance. This representation can be decoded for various tasks such as novel view rendering, 3D reconstruction, and stable grasp prediction.
  arXiv  Detail & Related papers  (2022-10-21T17:33:14Z)
• Vision Transformer for NeRF-Based View Synthesis from a Single Input Image [49.956005709863355]
  We propose to leverage both the global and local features to form an expressive 3D representation. To synthesize a novel view, we train a multilayer perceptron (MLP) network conditioned on the learned 3D representation to perform volume rendering. Our method can render novel views from only a single input image and generalize across multiple object categories using a single model.
  arXiv  Detail & Related papers  (2022-07-12T17:52:04Z)
• RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs [79.00855490550367]
  We show that NeRF can produce photorealistic renderings of unseen viewpoints when many input views are available. We address this by regularizing the geometry and appearance of patches rendered from unobserved viewpoints. Our model outperforms not only other methods that optimize over a single scene, but also conditional models that are extensively pre-trained on large multi-view datasets.
  arXiv  Detail & Related papers  (2021-12-01T18:59:46Z)
• pixelNeRF: Neural Radiance Fields from One or Few Images [20.607712035278315]
  pixelNeRF is a learning framework that predicts a continuous neural scene representation conditioned on one or few input images. We conduct experiments on ShapeNet benchmarks for single image novel view synthesis tasks with held-out objects. In all cases, pixelNeRF outperforms current state-of-the-art baselines for novel view synthesis and single image 3D reconstruction.
  arXiv  Detail & Related papers  (2020-12-03T18:59:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.