FBINeRF: Feature-Based Integrated Recurrent Network for Pinhole and Fisheye Neural Radiance Fields

• URL: http://arxiv.org/abs/2408.01878v1
• Date: Sat, 3 Aug 2024 23:11:20 GMT
• Title: FBINeRF: Feature-Based Integrated Recurrent Network for Pinhole and Fisheye Neural Radiance Fields
• Authors: Yifan Wu, Tianyi Cheng, Peixu Xin, Janusz Konrad,
• Abstract summary: We propose adaptive GRUs with a flexible bundle-adjustment method adapted to radial distortions. We show high-fidelity results for both pinhole-camera and fisheye-camera NeRFs.
• Score: 13.014637091971842
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Previous studies aiming to optimize and bundle-adjust camera poses using Neural Radiance Fields (NeRFs), such as BARF and DBARF, have demonstrated impressive capabilities in 3D scene reconstruction. However, these approaches have been designed for pinhole-camera pose optimization and do not perform well under radial image distortions such as those in fisheye cameras. Furthermore, inaccurate depth initialization in DBARF results in erroneous geometric information affecting the overall convergence and quality of results. In this paper, we propose adaptive GRUs with a flexible bundle-adjustment method adapted to radial distortions and incorporate feature-based recurrent neural networks to generate continuous novel views from fisheye datasets. Other NeRF methods for fisheye images, such as SCNeRF and OMNI-NeRF, use projected ray distance loss for distorted pose refinement, causing severe artifacts, long rendering time, and are difficult to use in downstream tasks, where the dense voxel representation generated by a NeRF method needs to be converted into a mesh representation. We also address depth initialization issues by adding MiDaS-based depth priors for pinhole images. Through extensive experiments, we demonstrate the generalization capacity of FBINeRF and show high-fidelity results for both pinhole-camera and fisheye-camera NeRFs.

Related papers

• TD-NeRF: Novel Truncated Depth Prior for Joint Camera Pose and Neural Radiance Field Optimization [19.73020713365866]
  The reliance on accurate camera poses is a significant barrier to the widespread deployment of Neural Radiance Fields (NeRF) models for 3D reconstruction and SLAM tasks. The existing method introduces monocular depth priors to jointly optimize the camera poses and NeRF, which fails to fully exploit the depth priors and neglects the impact of their inherent noise. We propose Truncated Depth NeRF (TD-NeRF), a novel approach that enables training NeRF from unknown camera poses - by jointly optimizing learnable parameters of the radiance field and camera poses.
  arXiv  Detail & Related papers  (2024-05-11T14:57:42Z)
• CF-NeRF: Camera Parameter Free Neural Radiance Fields with Incremental Learning [23.080474939586654]
  We propose a novel underlinecamera parameter underlinefree neural radiance field (CF-NeRF) CF-NeRF incrementally reconstructs 3D representations and recovers the camera parameters inspired by incremental structure from motion. Results demonstrate that CF-NeRF is robust to camera rotation and achieves state-of-the-art results without providing prior information and constraints.
  arXiv  Detail & Related papers  (2023-12-14T09:09:31Z)
• DaRF: Boosting Radiance Fields from Sparse Inputs with Monocular Depth Adaptation [31.655818586634258]
  We propose a novel framework, dubbed D"aRF, that achieves robust NeRF reconstruction with a handful of real-world images. Our framework imposes the MDE network's powerful geometry prior to NeRF representation at both seen and unseen viewpoints. In addition, we overcome the ambiguity problems of monocular depths through patch-wise scale-shift fitting and geometry distillation.
  arXiv  Detail & Related papers  (2023-05-30T16:46:41Z)
• Clean-NeRF: Reformulating NeRF to account for View-Dependent Observations [67.54358911994967]
  This paper proposes Clean-NeRF for accurate 3D reconstruction and novel view rendering in complex scenes. Clean-NeRF can be implemented as a plug-in that can immediately benefit existing NeRF-based methods without additional input.
  arXiv  Detail & Related papers  (2023-03-26T12:24:31Z)
• DBARF: Deep Bundle-Adjusting Generalizable Neural Radiance Fields [75.35416391705503]
  Recent works such as BARF and GARF can adjust camera poses with neural radiance fields (NeRF) which is based on coordinate-MLPs. Despite the impressive results, these methods cannot be applied to Generalizable NeRFs (GeNeRFs) which require image feature extractions. In this work, we first analyze the difficulties of jointly optimizing camera poses with GeNeRFs, and then further propose our DBARF to tackle these issues.
  arXiv  Detail & Related papers  (2023-03-25T14:18:30Z)
• NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusion [107.67277084886929]
  Novel view synthesis from a single image requires inferring occluded regions of objects and scenes whilst simultaneously maintaining semantic and physical consistency with the input. We propose NerfDiff, which addresses this issue by distilling the knowledge of a 3D-aware conditional diffusion model (CDM) into NeRF through synthesizing and refining a set of virtual views at test time. We further propose a novel NeRF-guided distillation algorithm that simultaneously generates 3D consistent virtual views from the CDM samples, and finetunes the NeRF based on the improved virtual views.
  arXiv  Detail & Related papers  (2023-02-20T17:12:00Z)
• Aug-NeRF: Training Stronger Neural Radiance Fields with Triple-Level Physically-Grounded Augmentations [111.08941206369508]
  We propose Augmented NeRF (Aug-NeRF), which for the first time brings the power of robust data augmentations into regularizing the NeRF training. Our proposal learns to seamlessly blend worst-case perturbations into three distinct levels of the NeRF pipeline. Aug-NeRF effectively boosts NeRF performance in both novel view synthesis and underlying geometry reconstruction.
  arXiv  Detail & Related papers  (2022-07-04T02:27:07Z)
• Depth-supervised NeRF: Fewer Views and Faster Training for Free [69.34556647743285]
  DS-NeRF (Depth-supervised Neural Radiance Fields) is a loss for learning fields that takes advantage of readily-available depth supervision. We show that our loss is compatible with other recently proposed NeRF methods, demonstrating that depth is a cheap and easily digestible supervisory signal.
  arXiv  Detail & Related papers  (2021-07-06T17:58:35Z)
• iNeRF: Inverting Neural Radiance Fields for Pose Estimation [68.91325516370013]
  We present iNeRF, a framework that performs mesh-free pose estimation by "inverting" a Neural RadianceField (NeRF) NeRFs have been shown to be remarkably effective for the task of view synthesis.
  arXiv  Detail & Related papers  (2020-12-10T18:36:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.