ManifoldNeRF: View-dependent Image Feature Supervision for Few-shot Neural Radiance Fields

• URL: http://arxiv.org/abs/2310.13670v1
• Date: Fri, 20 Oct 2023 17:13:52 GMT
• Title: ManifoldNeRF: View-dependent Image Feature Supervision for Few-shot Neural Radiance Fields
• Authors: Daiju Kanaoka, Motoharu Sonogashira, Hakaru Tamukoh, Yasutomo Kawanishi
• Abstract summary: DietNeRF is an extension of Neural Radiance Fields (NeRF) DietNeRF assumes that a pre-trained feature extractor should output the same feature even if input images are captured at different viewpoints. We propose ManifoldNeRF, a method for supervising feature vectors at unknown viewpoints.
• Score: 1.8512070255576754
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Novel view synthesis has recently made significant progress with the advent of Neural Radiance Fields (NeRF). DietNeRF is an extension of NeRF that aims to achieve this task from only a few images by introducing a new loss function for unknown viewpoints with no input images. The loss function assumes that a pre-trained feature extractor should output the same feature even if input images are captured at different viewpoints since the images contain the same object. However, while that assumption is ideal, in reality, it is known that as viewpoints continuously change, also feature vectors continuously change. Thus, the assumption can harm training. To avoid this harmful training, we propose ManifoldNeRF, a method for supervising feature vectors at unknown viewpoints using interpolated features from neighboring known viewpoints. Since the method provides appropriate supervision for each unknown viewpoint by the interpolated features, the volume representation is learned better than DietNeRF. Experimental results show that the proposed method performs better than others in a complex scene. We also experimented with several subsets of viewpoints from a set of viewpoints and identified an effective set of viewpoints for real environments. This provided a basic policy of viewpoint patterns for real-world application. The code is available at https://github.com/haganelego/ManifoldNeRF_BMVC2023

Related papers

• Re-Nerfing: Improving Novel View Synthesis through Novel View Synthesis [80.3686833921072]
  Recent neural rendering and reconstruction techniques, such as NeRFs or Gaussian Splatting, have shown remarkable novel view synthesis capabilities. With fewer images available, these methods start to fail since they can no longer correctly triangulate the underlying 3D geometry. We propose Re-Nerfing, a simple and general add-on approach that leverages novel view synthesis itself to tackle this problem.
  arXiv  Detail & Related papers  (2023-12-04T18:56:08Z)
• SimpleNeRF: Regularizing Sparse Input Neural Radiance Fields with Simpler Solutions [6.9980855647933655]
  supervising the depth estimated by the NeRF helps train it effectively with fewer views. We design augmented models that encourage simpler solutions by exploring the role of positional encoding and view-dependent radiance. We achieve state-of-the-art view-synthesis performance on two popular datasets by employing the above regularizations.
  arXiv  Detail & Related papers  (2023-09-07T18:02:57Z)
• ViP-NeRF: Visibility Prior for Sparse Input Neural Radiance Fields [9.67057831710618]
  Training neural radiance fields (NeRFs) on sparse input views leads to overfitting and incorrect scene depth estimation. We reformulate the NeRF to also directly output the visibility of a 3D point from a given viewpoint to reduce the training time with the visibility constraint. Our model outperforms the competing sparse input NeRF models including those that use learned priors.
  arXiv  Detail & Related papers  (2023-04-28T18:26:23Z)
• X-NeRF: Explicit Neural Radiance Field for Multi-Scene 360$^{\circ} $ Insufficient RGB-D Views [49.55319833743988]
  This paper focuses on a rarely discussed but important setting: can we train one model that can represent multiple scenes? We refer insufficient views to few extremely sparse and almost non-overlapping views. X-NeRF, a fully explicit approach which learns a general scene completion process instead of a coordinate-based mapping, is proposed.
  arXiv  Detail & Related papers  (2022-10-11T04:29:26Z)
• VM-NeRF: Tackling Sparsity in NeRF with View Morphing [19.418298933260953]
  NeRF aims to learn a continuous neural scene representation by using a finite set of input images taken from various viewpoints. This paper introduces a novel method to generate geometrically consistent image transitions between viewpoints using View Morphing.
  arXiv  Detail & Related papers  (2022-10-09T09:59:46Z)
• Vision Transformer for NeRF-Based View Synthesis from a Single Input Image [49.956005709863355]
  We propose to leverage both the global and local features to form an expressive 3D representation. To synthesize a novel view, we train a multilayer perceptron (MLP) network conditioned on the learned 3D representation to perform volume rendering. Our method can render novel views from only a single input image and generalize across multiple object categories using a single model.
  arXiv  Detail & Related papers  (2022-07-12T17:52:04Z)
• Ray Priors through Reprojection: Improving Neural Radiance Fields for Novel View Extrapolation [35.47411859184933]
  We study the novel view extrapolation setting that (1) the training images can well describe an object, and (2) there is a notable discrepancy between the training and test viewpoints' distributions. We propose a random ray casting policy that allows training unseen views using seen views. A ray atlas pre-computed from the observed rays' viewing directions could further enhance the rendering quality for extrapolated views.
  arXiv  Detail & Related papers  (2022-05-12T07:21:17Z)
• View Synthesis with Sculpted Neural Points [64.40344086212279]
  Implicit neural representations have achieved impressive visual quality but have drawbacks in computational efficiency. We propose a new approach that performs view synthesis using point clouds. It is the first point-based method to achieve better visual quality than NeRF while being more than 100x faster in rendering speed.
  arXiv  Detail & Related papers  (2022-05-12T03:54:35Z)
• Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis [86.38901313994734]
  We present DietNeRF, a 3D neural scene representation estimated from a few images. NeRF learns a continuous volumetric representation of a scene through multi-view consistency. We introduce an auxiliary semantic consistency loss that encourages realistic renderings at novel poses.
  arXiv  Detail & Related papers  (2021-04-01T17:59:31Z)
• Data Augmentation for Object Detection via Differentiable Neural Rendering [71.00447761415388]
  It is challenging to train a robust object detector when annotated data is scarce. Existing approaches to tackle this problem include semi-supervised learning that interpolates labeled data from unlabeled data. We introduce an offline data augmentation method for object detection, which semantically interpolates the training data with novel views.
  arXiv  Detail & Related papers  (2021-03-04T06:31:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.