Structure-Aware NeRF without Posed Camera via Epipolar Constraint

• URL: http://arxiv.org/abs/2210.00183v1
• Date: Sat, 1 Oct 2022 03:57:39 GMT
• Title: Structure-Aware NeRF without Posed Camera via Epipolar Constraint
• Authors: Shu Chen, Yang Zhang, Yaxin Xu, and Beiji Zou
• Abstract summary: The neural radiance field (NeRF) for realistic novel view synthesis requires camera poses to be pre-acquired. We integrate the pose extraction and view synthesis into a single end-to-end procedure so they can benefit from each other.
• Score: 8.115535686311249
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The neural radiance field (NeRF) for realistic novel view synthesis requires camera poses to be pre-acquired by a structure-from-motion (SfM) approach. This two-stage strategy is not convenient to use and degrades the performance because the error in the pose extraction can propagate to the view synthesis. We integrate the pose extraction and view synthesis into a single end-to-end procedure so they can benefit from each other. For training NeRF models, only RGB images are given, without pre-known camera poses. The camera poses are obtained by the epipolar constraint in which the identical feature in different views has the same world coordinates transformed from the local camera coordinates according to the extracted poses. The epipolar constraint is jointly optimized with pixel color constraint. The poses are represented by a CNN-based deep network, whose input is the related frames. This joint optimization enables NeRF to be aware of the scene's structure that has an improved generalization performance. Extensive experiments on a variety of scenes demonstrate the effectiveness of the proposed approach. Code is available at https://github.com/XTU-PR-LAB/SaNerf.

Related papers

• CT-NeRF: Incremental Optimizing Neural Radiance Field and Poses with Complex Trajectory [12.460959809597213]
  We propose CT-NeRF, an incremental reconstruction optimization pipeline using only RGB images without pose and depth input. We evaluate the performance of CT-NeRF on two real-world datasets, NeRFBuster and Free-Dataset.
  arXiv  Detail & Related papers  (2024-04-22T06:07:06Z)
• COLMAP-Free 3D Gaussian Splatting [88.420322646756]
  We propose a novel method to perform novel view synthesis without any SfM preprocessing. We process the input frames in a sequential manner and progressively grow the 3D Gaussians set by taking one input frame at a time. Our method significantly improves over previous approaches in view synthesis and camera pose estimation under large motion changes.
  arXiv  Detail & Related papers  (2023-12-12T18:39:52Z)
• Pose-Free Generalizable Rendering Transformer [72.47072706742065]
  PF-GRT is a Pose-Free framework for Generalizable Rendering Transformer. PF-GRT is parameterized using a local relative coordinate system. Experiments with zero-shot rendering on datasets reveal that it produces superior quality in generating photo-realistic images.
  arXiv  Detail & Related papers  (2023-10-05T17:24:36Z)
• DBARF: Deep Bundle-Adjusting Generalizable Neural Radiance Fields [75.35416391705503]
  Recent works such as BARF and GARF can adjust camera poses with neural radiance fields (NeRF) which is based on coordinate-MLPs. Despite the impressive results, these methods cannot be applied to Generalizable NeRFs (GeNeRFs) which require image feature extractions. In this work, we first analyze the difficulties of jointly optimizing camera poses with GeNeRFs, and then further propose our DBARF to tackle these issues.
  arXiv  Detail & Related papers  (2023-03-25T14:18:30Z)
• SPARF: Neural Radiance Fields from Sparse and Noisy Poses [58.528358231885846]
  We introduce Sparse Pose Adjusting Radiance Field (SPARF) to address the challenge of novel-view synthesis. Our approach exploits multi-view geometry constraints in order to jointly learn the NeRF and refine the camera poses.
  arXiv  Detail & Related papers  (2022-11-21T18:57:47Z)
• BARF: Bundle-Adjusting Neural Radiance Fields [104.97810696435766]
  We propose Bundle-Adjusting Neural Radiance Fields (BARF) for training NeRF from imperfect camera poses. BARF can effectively optimize the neural scene representations and resolve large camera pose misalignment at the same time. This enables view synthesis and localization of video sequences from unknown camera poses, opening up new avenues for visual localization systems.
  arXiv  Detail & Related papers  (2021-04-13T17:59:51Z)
• GNeRF: GAN-based Neural Radiance Field without Posed Camera [67.80805274569354]
  We introduce GNeRF, a framework to marry Generative Adversarial Networks (GAN) with Neural Radiance Field reconstruction for the complex scenarios with unknown and even randomly camera poses. Our approach outperforms the baselines favorably in those scenes with repeated patterns or even low textures that are regarded as extremely challenging before.
  arXiv  Detail & Related papers  (2021-03-29T13:36:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.