Non-uniform Sampling Strategies for NeRF on 360{\textdegree} images

• URL: http://arxiv.org/abs/2212.03635v1
• Date: Wed, 7 Dec 2022 13:48:16 GMT
• Title: Non-uniform Sampling Strategies for NeRF on 360{\textdegree} images
• Authors: Takashi Otonari, Satoshi Ikehata, Kiyoharu Aizawa
• Abstract summary: This study proposes two novel techniques that effectively build NeRF for 360textdegree omnidirectional images. We propose two non-uniform ray sampling schemes for NeRF to suit 360textdegree images. We show that our proposed method enhances the quality of real-world scenes in 360textdegree images.
• Score: 40.02598009484401
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In recent years, the performance of novel view synthesis using perspective images has dramatically improved with the advent of neural radiance fields (NeRF). This study proposes two novel techniques that effectively build NeRF for 360{\textdegree} omnidirectional images. Due to the characteristics of a 360{\textdegree} image of ERP format that has spatial distortion in their high latitude regions and a 360{\textdegree} wide viewing angle, NeRF's general ray sampling strategy is ineffective. Hence, the view synthesis accuracy of NeRF is limited and learning is not efficient. We propose two non-uniform ray sampling schemes for NeRF to suit 360{\textdegree} images - distortion-aware ray sampling and content-aware ray sampling. We created an evaluation dataset Synth360 using Replica and SceneCity models of indoor and outdoor scenes, respectively. In experiments, we show that our proposal successfully builds 360{\textdegree} image NeRF in terms of both accuracy and efficiency. The proposal is widely applicable to advanced variants of NeRF. DietNeRF, AugNeRF, and NeRF++ combined with the proposed techniques further improve the performance. Moreover, we show that our proposed method enhances the quality of real-world scenes in 360{\textdegree} images. Synth360: https://drive.google.com/drive/folders/1suL9B7DO2no21ggiIHkH3JF3OecasQLb.

Related papers

• ZeroRF: Fast Sparse View 360{\deg} Reconstruction with Zero Pretraining [28.03297623406931]
  Current breakthroughs like Neural Radiance Fields (NeRF) have demonstrated high-fidelity image synthesis but struggle with sparse input views. We propose ZeroRF, whose key idea is to integrate a tailored Deep Image Prior into a factorized NeRF representation. Unlike traditional methods, ZeroRF parametrizes feature grids with a neural network generator, enabling efficient sparse view 360deg reconstruction.
  arXiv  Detail & Related papers  (2023-12-14T18:59:32Z)
• NeO 360: Neural Fields for Sparse View Synthesis of Outdoor Scenes [59.15910989235392]
  We introduce NeO 360, Neural fields for sparse view synthesis of outdoor scenes. NeO 360 is a generalizable method that reconstructs 360deg scenes from a single or a few posed RGB images. Our representation combines the best of both voxel-based and bird's-eye-view (BEV) representations.
  arXiv  Detail & Related papers  (2023-08-24T17:59:50Z)
• Multi-Space Neural Radiance Fields [74.46513422075438]
  Existing Neural Radiance Fields (NeRF) methods suffer from the existence of reflective objects. We propose a multi-space neural radiance field (MS-NeRF) that represents the scene using a group of feature fields in parallel sub-spaces. Our approach significantly outperforms the existing single-space NeRF methods for rendering high-quality scenes.
  arXiv  Detail & Related papers  (2023-05-07T13:11:07Z)
• Improving Neural Radiance Fields with Depth-aware Optimization for Novel View Synthesis [12.3338393483795]
  We propose SfMNeRF, a method to better synthesize novel views as well as reconstruct the 3D-scene geometry. SfMNeRF employs the epipolar, photometric consistency, depth smoothness, and position-of-matches constraints to explicitly reconstruct the 3D-scene structure. Experiments on two public datasets demonstrate that SfMNeRF surpasses state-of-the-art approaches.
  arXiv  Detail & Related papers  (2023-04-11T13:37:17Z)
• SPARF: Large-Scale Learning of 3D Sparse Radiance Fields from Few Input Images [62.64942825962934]
  We present SPARF, a large-scale ShapeNet-based synthetic dataset for novel view synthesis. We propose a novel pipeline (SuRFNet) that learns to generate sparse voxel radiance fields from only few views. SuRFNet employs partial SRFs from few/one images and a specialized SRF loss to learn to generate high-quality sparse voxel radiance fields.
  arXiv  Detail & Related papers  (2022-12-18T14:56:22Z)
• Enhancement of Novel View Synthesis Using Omnidirectional Image Completion [61.78187618370681]
  We present a method for synthesizing novel views from a single 360-degree RGB-D image based on the neural radiance field (NeRF) Experiments demonstrated that the proposed method can synthesize plausible novel views while preserving the features of the scene for both artificial and real-world data.
  arXiv  Detail & Related papers  (2022-03-18T13:49:25Z)
• NeRF++: Analyzing and Improving Neural Radiance Fields [117.73411181186088]
  Neural Radiance Fields (NeRF) achieve impressive view synthesis results for a variety of capture settings. NeRF fits multi-layer perceptrons representing view-invariant opacity and view-dependent color volumes to a set of training images. We address a parametrization issue involved in applying NeRF to 360 captures of objects within large-scale, 3D scenes.
  arXiv  Detail & Related papers  (2020-10-15T03:24:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.