LENS: Localization enhanced by NeRF synthesis
- URL: http://arxiv.org/abs/2110.06558v1
- Date: Wed, 13 Oct 2021 08:15:08 GMT
- Title: LENS: Localization enhanced by NeRF synthesis
- Authors: Arthur Moreau, Nathan Piasco, Dzmitry Tsishkou, Bogdan Stanciulescu,
Arnaud de La Fortelle
- Abstract summary: We demonstrate improvement of camera pose regression thanks to an additional synthetic dataset rendered by the NeRF class of algorithm.
We further improved localization accuracy of pose regressors using synthesized realistic and geometry consistent images as data augmentation during training.
- Score: 3.4386226615580107
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: Neural Radiance Fields (NeRF) have recently demonstrated photo-realistic
results for the task of novel view synthesis. In this paper, we propose to
apply novel view synthesis to the robot relocalization problem: we demonstrate
improvement of camera pose regression thanks to an additional synthetic dataset
rendered by the NeRF class of algorithm. To avoid spawning novel views in
irrelevant places we selected virtual camera locations from NeRF internal
representation of the 3D geometry of the scene. We further improved
localization accuracy of pose regressors using synthesized realistic and
geometry consistent images as data augmentation during training. At the time of
publication, our approach improved state of the art with a 60% lower error on
Cambridge Landmarks and 7-scenes datasets. Hence, the resulting accuracy
becomes comparable to structure-based methods, without any architecture
modification or domain adaptation constraints. Since our method allows almost
infinite generation of training data, we investigated limitations of camera
pose regression depending on size and distribution of data used for training on
public benchmarks. We concluded that pose regression accuracy is mostly bounded
by relatively small and biased datasets rather than capacity of the pose
regression model to solve the localization task.
Related papers
- No Pose, No Problem: Surprisingly Simple 3D Gaussian Splats from Sparse Unposed Images [100.80376573969045]
NoPoSplat is a feed-forward model capable of reconstructing 3D scenes parameterized by 3D Gaussians from multi-view images.
Our model achieves real-time 3D Gaussian reconstruction during inference.
This work makes significant advances in pose-free generalizable 3D reconstruction and demonstrates its applicability to real-world scenarios.
arXiv Detail & Related papers (2024-10-31T17:58:22Z) - SG-NeRF: Neural Surface Reconstruction with Scene Graph Optimization [16.460851701725392]
We present a novel approach that optimize radiance fields with scene graphs to mitigate the influence of outlier poses.
Our method incorporates an adaptive inlier-outlier confidence estimation scheme based on scene graphs.
We also introduce an effective intersection-over-union (IoU) loss to optimize the camera pose and surface geometry.
arXiv Detail & Related papers (2024-07-17T15:50:17Z) - Leveraging Neural Radiance Field in Descriptor Synthesis for Keypoints Scene Coordinate Regression [1.2974519529978974]
This paper introduces a pipeline for keypoint descriptor synthesis using Neural Radiance Field (NeRF)
generating novel poses and feeding them into a trained NeRF model to create new views, our approach enhances the KSCR's capabilities in data-scarce environments.
The proposed system could significantly improve localization accuracy by up to 50% and cost only a fraction of time for data synthesis.
arXiv Detail & Related papers (2024-03-15T13:40:37Z) - PNeRFLoc: Visual Localization with Point-based Neural Radiance Fields [54.8553158441296]
We propose a novel visual localization framework, ie, PNeRFLoc, based on a unified point-based representation.
On the one hand, PNeRFLoc supports the initial pose estimation by matching 2D and 3D feature points.
On the other hand, it also enables pose refinement with novel view synthesis using rendering-based optimization.
arXiv Detail & Related papers (2023-12-17T08:30:00Z) - Re-Nerfing: Improving Novel View Synthesis through Novel View Synthesis [80.3686833921072]
Recent neural rendering and reconstruction techniques, such as NeRFs or Gaussian Splatting, have shown remarkable novel view synthesis capabilities.
With fewer images available, these methods start to fail since they can no longer correctly triangulate the underlying 3D geometry.
We propose Re-Nerfing, a simple and general add-on approach that leverages novel view synthesis itself to tackle this problem.
arXiv Detail & Related papers (2023-12-04T18:56:08Z) - Leveraging Neural Radiance Fields for Uncertainty-Aware Visual
Localization [56.95046107046027]
We propose to leverage Neural Radiance Fields (NeRF) to generate training samples for scene coordinate regression.
Despite NeRF's efficiency in rendering, many of the rendered data are polluted by artifacts or only contain minimal information gain.
arXiv Detail & Related papers (2023-10-10T20:11:13Z) - Neural Refinement for Absolute Pose Regression with Feature Synthesis [33.2608395824548]
Absolute Pose Regression (APR) methods use deep neural networks to directly regress camera poses from RGB images.
In this work, we propose a test-time refinement pipeline that leverages implicit geometric constraints.
We also introduce a novel Neural Feature Synthesizer (NeFeS) model, which encodes 3D geometric features during training and directly renders dense novel view features at test time to refine APR methods.
arXiv Detail & Related papers (2023-03-17T16:10:50Z) - Shape, Pose, and Appearance from a Single Image via Bootstrapped
Radiance Field Inversion [54.151979979158085]
We introduce a principled end-to-end reconstruction framework for natural images, where accurate ground-truth poses are not available.
We leverage an unconditional 3D-aware generator, to which we apply a hybrid inversion scheme where a model produces a first guess of the solution.
Our framework can de-render an image in as few as 10 steps, enabling its use in practical scenarios.
arXiv Detail & Related papers (2022-11-21T17:42:42Z) - DFNet: Enhance Absolute Pose Regression with Direct Feature Matching [16.96571417692014]
We introduce a camera relocalization pipeline that combines absolute pose regression (APR) and direct feature matching.
We show that our method achieves a state-of-the-art accuracy by outperforming existing single-image APR methods by as much as 56%, comparable to 3D structure-based methods.
arXiv Detail & Related papers (2022-04-01T16:39:16Z) - BARF: Bundle-Adjusting Neural Radiance Fields [104.97810696435766]
We propose Bundle-Adjusting Neural Radiance Fields (BARF) for training NeRF from imperfect camera poses.
BARF can effectively optimize the neural scene representations and resolve large camera pose misalignment at the same time.
This enables view synthesis and localization of video sequences from unknown camera poses, opening up new avenues for visual localization systems.
arXiv Detail & Related papers (2021-04-13T17:59:51Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.