NEWTON: Neural View-Centric Mapping for On-the-Fly Large-Scale SLAM
- URL: http://arxiv.org/abs/2303.13654v2
- Date: Wed, 29 Mar 2023 15:12:24 GMT
- Title: NEWTON: Neural View-Centric Mapping for On-the-Fly Large-Scale SLAM
- Authors: Hidenobu Matsuki, Keisuke Tateno, Michael Niemeyer, Federico Tombari
- Abstract summary: Newton is a view-centric mapping method that dynamically constructs neural fields based on run-time observation.
Our method enables camera pose updates using loop closures and scene boundary updates by representing the scene with multiple neural fields.
The experimental results demonstrate the superior performance of our method over existing world-centric neural field-based SLAM systems.
- Score: 51.21564182169607
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Neural field-based 3D representations have recently been adopted in many
areas including SLAM systems. Current neural SLAM or online mapping systems
lead to impressive results in the presence of simple captures, but they rely on
a world-centric map representation as only a single neural field model is used.
To define such a world-centric representation, accurate and static prior
information about the scene, such as its boundaries and initial camera poses,
are required. However, in real-time and on-the-fly scene capture applications,
this prior knowledge cannot be assumed as fixed or static, since it dynamically
changes and it is subject to significant updates based on run-time
observations. Particularly in the context of large-scale mapping, significant
camera pose drift is inevitable, necessitating the correction via loop closure.
To overcome this limitation, we propose NEWTON, a view-centric mapping method
that dynamically constructs neural fields based on run-time observation. In
contrast to prior works, our method enables camera pose updates using loop
closures and scene boundary updates by representing the scene with multiple
neural fields, where each is defined in a local coordinate system of a selected
keyframe. The experimental results demonstrate the superior performance of our
method over existing world-centric neural field-based SLAM systems, in
particular for large-scale scenes subject to camera pose updates.
Related papers
- MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion [118.74385965694694]
We present Motion DUSt3R (MonST3R), a novel geometry-first approach that directly estimates per-timestep geometry from dynamic scenes.
By simply estimating a pointmap for each timestep, we can effectively adapt DUST3R's representation, previously only used for static scenes, to dynamic scenes.
We show that by posing the problem as a fine-tuning task, identifying several suitable datasets, and strategically training the model on this limited data, we can surprisingly enable the model to handle dynamics.
arXiv Detail & Related papers (2024-10-04T18:00:07Z) - OmniLocalRF: Omnidirectional Local Radiance Fields from Dynamic Videos [14.965321452764355]
We introduce a new approach called Omnidirectional Local Radiance Fields (OmniLocalRF) that can render static-only scene views.
Our approach combines the principles of local radiance fields with the bidirectional optimization of omnidirectional rays.
Our experiments validate that OmniLocalRF outperforms existing methods in both qualitative and quantitative metrics.
arXiv Detail & Related papers (2024-03-31T12:55:05Z) - NeSLAM: Neural Implicit Mapping and Self-Supervised Feature Tracking With Depth Completion and Denoising [23.876281686625134]
We present NeSLAM, a framework that achieves accurate and dense depth estimation, robust camera tracking, and realistic synthesis of novel views.
Experiments on various indoor datasets demonstrate the effectiveness and accuracy of the system in reconstruction, tracking quality, and novel view synthesis.
arXiv Detail & Related papers (2024-03-29T07:59:37Z) - DNS SLAM: Dense Neural Semantic-Informed SLAM [92.39687553022605]
DNS SLAM is a novel neural RGB-D semantic SLAM approach featuring a hybrid representation.
Our method integrates multi-view geometry constraints with image-based feature extraction to improve appearance details.
Our experimental results achieve state-of-the-art performance on both synthetic data and real-world data tracking.
arXiv Detail & Related papers (2023-11-30T21:34:44Z) - Learning Robust Multi-Scale Representation for Neural Radiance Fields
from Unposed Images [65.41966114373373]
We present an improved solution to the neural image-based rendering problem in computer vision.
The proposed approach could synthesize a realistic image of the scene from a novel viewpoint at test time.
arXiv Detail & Related papers (2023-11-08T08:18:23Z) - Nerfels: Renderable Neural Codes for Improved Camera Pose Estimation [21.111919718001907]
Our proposed 3D scene representation, Nerfels, is locally dense yet globally sparse.
We adopt a feature-driven approach for representing scene-agnostic, local 3D patches with renderable codes.
Our model can be incorporated to existing state-of-the-art hand-crafted and learned local feature estimators, yielding improved performance when evaluating on ScanNet for wide camera baseline scenarios.
arXiv Detail & Related papers (2022-06-04T06:29:46Z) - BARF: Bundle-Adjusting Neural Radiance Fields [104.97810696435766]
We propose Bundle-Adjusting Neural Radiance Fields (BARF) for training NeRF from imperfect camera poses.
BARF can effectively optimize the neural scene representations and resolve large camera pose misalignment at the same time.
This enables view synthesis and localization of video sequences from unknown camera poses, opening up new avenues for visual localization systems.
arXiv Detail & Related papers (2021-04-13T17:59:51Z) - D-NeRF: Neural Radiance Fields for Dynamic Scenes [72.75686949608624]
We introduce D-NeRF, a method that extends neural radiance fields to a dynamic domain.
D-NeRF reconstructs images of objects under rigid and non-rigid motions from a camera moving around the scene.
We demonstrate the effectiveness of our approach on scenes with objects under rigid, articulated and non-rigid motions.
arXiv Detail & Related papers (2020-11-27T19:06:50Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.