TivNe-SLAM: Dynamic Mapping and Tracking via Time-Varying Neural Radiance Fields
- URL: http://arxiv.org/abs/2310.18917v4
- Date: Mon, 18 Mar 2024 03:37:31 GMT
- Title: TivNe-SLAM: Dynamic Mapping and Tracking via Time-Varying Neural Radiance Fields
- Authors: Chengyao Duan, Zhiliu Yang,
- Abstract summary: We propose a time-varying representation to track and reconstruct dynamic scenes.
For the tracking process, all input images are uniformly sampled, then progressively trained in a self-supervised paradigm.
Experiments validate our method when compared to existing state-of-the-art NeRF-based methods.
- Score: 0.1227734309612871
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Previous attempts to integrate Neural Radiance Fields (NeRF) into the Simultaneous Localization and Mapping (SLAM) framework either rely on the assumption of static scenes or require the ground truth camera poses, which impedes their application in real-world scenarios. In this paper, we propose a time-varying representation to track and reconstruct the dynamic scenes. Firstly, two processes, tracking process and mapping process, are simultaneously maintained in our framework. For the tracking process, all input images are uniformly sampled, then progressively trained in a self-supervised paradigm. For the mapping process, we leverage motion masks to distinguish dynamic objects from static background, and sample more pixels from dynamic areas. Secondly, the parameter optimization for both processes consists of two stages: the first stage associates time with 3D positions to convert the deformation field to the canonical field. And the second stage associates time with the embeddings of canonical field to obtain colors and Signed Distance Function (SDF). Lastly, we propose a novel keyframe selection strategy based on the overlapping rate. We evaluate our approach on two synthetic datasets and one real-world dataset. And the experiments validate that our method achieves competitive results in both tracking and mapping when compared to existing state-of-the-art NeRF-based methods.
Related papers
- Hierarchical Temporal Context Learning for Camera-based Semantic Scene Completion [57.232688209606515]
We present HTCL, a novel Temporal Temporal Context Learning paradigm for improving camera-based semantic scene completion.
Our method ranks $1st$ on the Semantic KITTI benchmark and even surpasses LiDAR-based methods in terms of mIoU.
arXiv Detail & Related papers (2024-07-02T09:11:17Z) - NVFi: Neural Velocity Fields for 3D Physics Learning from Dynamic Videos [8.559809421797784]
We propose to simultaneously learn the geometry, appearance, and physical velocity of 3D scenes only from video frames.
We conduct extensive experiments on multiple datasets, demonstrating the superior performance of our method over all baselines.
arXiv Detail & Related papers (2023-12-11T14:07:31Z) - DNS SLAM: Dense Neural Semantic-Informed SLAM [92.39687553022605]
DNS SLAM is a novel neural RGB-D semantic SLAM approach featuring a hybrid representation.
Our method integrates multi-view geometry constraints with image-based feature extraction to improve appearance details.
Our experimental results achieve state-of-the-art performance on both synthetic data and real-world data tracking.
arXiv Detail & Related papers (2023-11-30T21:34:44Z) - EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via
Self-Supervision [85.17951804790515]
EmerNeRF is a simple yet powerful approach for learning spatial-temporal representations of dynamic driving scenes.
It simultaneously captures scene geometry, appearance, motion, and semantics via self-bootstrapping.
Our method achieves state-of-the-art performance in sensor simulation.
arXiv Detail & Related papers (2023-11-03T17:59:55Z) - TAPIR: Tracking Any Point with per-frame Initialization and temporal
Refinement [64.11385310305612]
We present a novel model for Tracking Any Point (TAP) that effectively tracks any queried point on any physical surface throughout a video sequence.
Our approach employs two stages: (1) a matching stage, which independently locates a suitable candidate point match for the query point on every other frame, and (2) a refinement stage, which updates both the trajectory and query features based on local correlations.
The resulting model surpasses all baseline methods by a significant margin on the TAP-Vid benchmark, as demonstrated by an approximate 20% absolute average Jaccard (AJ) improvement on DAVIS.
arXiv Detail & Related papers (2023-06-14T17:07:51Z) - 3D Multi-Object Tracking with Differentiable Pose Estimation [0.0]
We propose a novel approach for joint 3D multi-object tracking and reconstruction from RGB-D sequences in indoor environments.
We leverage those correspondences to inform a graph neural network to solve for the optimal, temporally-consistent 7-DoF pose trajectories of all objects.
Our method improves the accumulated MOTA score for all test sequences by 24.8% over existing state-of-the-art methods.
arXiv Detail & Related papers (2022-06-28T06:46:32Z) - Neural Deformable Voxel Grid for Fast Optimization of Dynamic View
Synthesis [63.25919018001152]
We propose a fast deformable radiance field method to handle dynamic scenes.
Our method achieves comparable performance to D-NeRF using only 20 minutes for training.
arXiv Detail & Related papers (2022-06-15T17:49:08Z) - TimeLens: Event-based Video Frame Interpolation [54.28139783383213]
We introduce Time Lens, a novel indicates equal contribution method that leverages the advantages of both synthesis-based and flow-based approaches.
We show an up to 5.21 dB improvement in terms of PSNR over state-of-the-art frame-based and event-based methods.
arXiv Detail & Related papers (2021-06-14T10:33:47Z) - CDN-MEDAL: Two-stage Density and Difference Approximation Framework for
Motion Analysis [3.337126420148156]
We propose a novel, two-stage method of change detection with two convolutional neural networks.
Our two-stage framework contains approximately 3.5K parameters in total but still maintains rapid convergence to intricate motion patterns.
arXiv Detail & Related papers (2021-06-07T16:39:42Z) - Deep Learning based Virtual Point Tracking for Real-Time Target-less
Dynamic Displacement Measurement in Railway Applications [0.0]
We propose virtual point tracking for real-time target-less dynamic displacement measurement, incorporating deep learning techniques and domain knowledge.
We demonstrate our approach for a railway application, where the lateral displacement of the wheel on the rail is measured during operation.
arXiv Detail & Related papers (2021-01-17T16:19:47Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.