CBARF: Cascaded Bundle-Adjusting Neural Radiance Fields from Imperfect Camera Poses

• URL: http://arxiv.org/abs/2310.09776v1
• Date: Sun, 15 Oct 2023 08:34:40 GMT
• Title: CBARF: Cascaded Bundle-Adjusting Neural Radiance Fields from Imperfect Camera Poses
• Authors: Hongyu Fu, Xin Yu, Lincheng Li, and Li Zhang
• Abstract summary: We propose a novel 3D reconstruction framework that enables simultaneous optimization of camera poses. In a nutshell, our framework optimize camera poses in a coarse-to-fine manner and then reconstructs scenes based on the rectified poses. Experimental results demonstrate that our CBARF model achieves state-of-the-art performance in both pose optimization and novel view synthesis.
• Score: 23.427859480410934
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing volumetric neural rendering techniques, such as Neural Radiance Fields (NeRF), face limitations in synthesizing high-quality novel views when the camera poses of input images are imperfect. To address this issue, we propose a novel 3D reconstruction framework that enables simultaneous optimization of camera poses, dubbed CBARF (Cascaded Bundle-Adjusting NeRF).In a nutshell, our framework optimizes camera poses in a coarse-to-fine manner and then reconstructs scenes based on the rectified poses. It is observed that the initialization of camera poses has a significant impact on the performance of bundle-adjustment (BA). Therefore, we cascade multiple BA modules at different scales to progressively improve the camera poses. Meanwhile, we develop a neighbor-replacement strategy to further optimize the results of BA in each stage. In this step, we introduce a novel criterion to effectively identify poorly estimated camera poses. Then we replace them with the poses of neighboring cameras, thus further eliminating the impact of inaccurate camera poses. Once camera poses have been optimized, we employ a density voxel grid to generate high-quality 3D reconstructed scenes and images in novel views. Experimental results demonstrate that our CBARF model achieves state-of-the-art performance in both pose optimization and novel view synthesis, especially in the existence of large camera pose noise.

Related papers

• Hybrid bundle-adjusting 3D Gaussians for view consistent rendering with pose optimization [2.8990883469500286]
  We introduce a hybrid bundle-adjusting 3D Gaussians model that enables view-consistent rendering with pose optimization. This model jointly extract image-based and neural 3D representations to simultaneously generate view-consistent images and camera poses within forward-facing scenes.
  arXiv  Detail & Related papers  (2024-10-17T07:13:00Z)
• Gaussian Splatting on the Move: Blur and Rolling Shutter Compensation for Natural Camera Motion [25.54868552979793]
  We present a method that adapts to camera motion and allows high-quality scene reconstruction with handheld video data. Our results with both synthetic and real data demonstrate superior performance in mitigating camera motion over existing methods.
  arXiv  Detail & Related papers  (2024-03-20T06:19:41Z)
• COLMAP-Free 3D Gaussian Splatting [88.420322646756]
  We propose a novel method to perform novel view synthesis without any SfM preprocessing. We process the input frames in a sequential manner and progressively grow the 3D Gaussians set by taking one input frame at a time. Our method significantly improves over previous approaches in view synthesis and camera pose estimation under large motion changes.
  arXiv  Detail & Related papers  (2023-12-12T18:39:52Z)
• Continuous Pose for Monocular Cameras in Neural Implicit Representation [65.40527279809474]
  In this paper, we showcase the effectiveness of optimizing monocular camera poses as a continuous function of time. We exploit the proposed method in four diverse experimental settings. Using the assumption of continuous motion, changes in pose may actually live in a manifold that has lower than 6 degrees of freedom (DOF) We call this low DOF motion representation as the emphintrinsic motion and use the approach in vSLAM settings, showing impressive camera tracking performance.
  arXiv  Detail & Related papers  (2023-11-28T13:14:58Z)
• ContraNeRF: 3D-Aware Generative Model via Contrastive Learning with Unsupervised Implicit Pose Embedding [40.36882490080341]
  We propose a novel 3D-aware GAN optimization technique through contrastive learning with implicit pose embeddings. We make the discriminator estimate a high-dimensional implicit pose embedding from a given image and perform contrastive learning on the pose embedding. The proposed approach can be employed for the dataset, where the canonical camera pose is ill-defined because it does not look up or estimate camera poses.
  arXiv  Detail & Related papers  (2023-04-27T07:53:13Z)
• Neural Lens Modeling [50.57409162437732]
  NeuroLens is a neural lens model for distortion and vignetting that can be used for point projection and ray casting. It can be used to perform pre-capture calibration using classical calibration targets, and can later be used to perform calibration or refinement during 3D reconstruction. The model generalizes across many lens types and is trivial to integrate into existing 3D reconstruction and rendering systems.
  arXiv  Detail & Related papers  (2023-04-10T20:09:17Z)
• Enhanced Stable View Synthesis [86.69338893753886]
  We introduce an approach to enhance the novel view synthesis from images taken from a freely moving camera. The introduced approach focuses on outdoor scenes where recovering accurate geometric scaffold and camera pose is challenging.
  arXiv  Detail & Related papers  (2023-03-30T01:53:14Z)
• SPARF: Neural Radiance Fields from Sparse and Noisy Poses [58.528358231885846]
  We introduce Sparse Pose Adjusting Radiance Field (SPARF) to address the challenge of novel-view synthesis. Our approach exploits multi-view geometry constraints in order to jointly learn the NeRF and refine the camera poses.
  arXiv  Detail & Related papers  (2022-11-21T18:57:47Z)
• GNeRF: GAN-based Neural Radiance Field without Posed Camera [67.80805274569354]
  We introduce GNeRF, a framework to marry Generative Adversarial Networks (GAN) with Neural Radiance Field reconstruction for the complex scenarios with unknown and even randomly camera poses. Our approach outperforms the baselines favorably in those scenes with repeated patterns or even low textures that are regarded as extremely challenging before.
  arXiv  Detail & Related papers  (2021-03-29T13:36:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.