Cross-Modal Semi-Dense 6-DoF Tracking of an Event Camera in Challenging
Conditions
- URL: http://arxiv.org/abs/2401.08043v1
- Date: Tue, 16 Jan 2024 01:48:45 GMT
- Title: Cross-Modal Semi-Dense 6-DoF Tracking of an Event Camera in Challenging
Conditions
- Authors: Yi-Fan Zuo, Wanting Xu, Xia Wang, Yifu Wang, Laurent Kneip
- Abstract summary: Event-based cameras are bio-inspired visual sensors that perform well in HDR conditions and have high temporal resolution.
The present work demonstrates the feasibility of purely event-based tracking if an alternative sensor is permitted for mapping.
The method relies on geometric 3D-2D registration of semi-dense maps and events, and achieves highly reliable and accurate cross-modal tracking results.
- Score: 29.608665442108727
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Vision-based localization is a cost-effective and thus attractive solution
for many intelligent mobile platforms. However, its accuracy and especially
robustness still suffer from low illumination conditions, illumination changes,
and aggressive motion. Event-based cameras are bio-inspired visual sensors that
perform well in HDR conditions and have high temporal resolution, and thus
provide an interesting alternative in such challenging scenarios. While purely
event-based solutions currently do not yet produce satisfying mapping results,
the present work demonstrates the feasibility of purely event-based tracking if
an alternative sensor is permitted for mapping. The method relies on geometric
3D-2D registration of semi-dense maps and events, and achieves highly reliable
and accurate cross-modal tracking results. Practically relevant scenarios are
given by depth camera-supported tracking or map-based localization with a
semi-dense map prior created by a regular image-based visual SLAM or
structure-from-motion system. Conventional edge-based 3D-2D alignment is
extended by a novel polarity-aware registration that makes use of signed
time-surface maps (STSM) obtained from event streams. We furthermore introduce
a novel culling strategy for occluded points. Both modifications increase the
speed of the tracker and its robustness against occlusions or large view-point
variations. The approach is validated on many real datasets covering the
above-mentioned challenging conditions, and compared against similar solutions
realised with regular cameras.
Related papers
- 3D Pose Estimation of Two Interacting Hands from a Monocular Event
Camera [59.846927201816776]
This paper introduces the first framework for 3D tracking of two fast-moving and interacting hands from a single monocular event camera.
Our approach tackles the left-right hand ambiguity with a novel semi-supervised feature-wise attention mechanism and integrates an intersection loss to fix hand collisions.
arXiv Detail & Related papers (2023-12-21T18:59:57Z) - EVI-SAM: Robust, Real-time, Tightly-coupled Event-Visual-Inertial State Estimation and 3D Dense Mapping [5.154689086578339]
We propose EVI-SAM to tackle the problem of 6 DoF pose tracking and 3D reconstruction using monocular event camera.
A novel event-based hybrid tracking framework is designed to estimate the pose, leveraging the robustness of feature matching and the precision of direct alignment.
To the best of our knowledge, this is the first non-learning work to realize event-based dense mapping.
arXiv Detail & Related papers (2023-12-19T07:39:45Z) - Implicit Event-RGBD Neural SLAM [54.74363487009845]
Implicit neural SLAM has achieved remarkable progress recently.
Existing methods face significant challenges in non-ideal scenarios.
We propose EN-SLAM, the first event-RGBD implicit neural SLAM framework.
arXiv Detail & Related papers (2023-11-18T08:48:58Z) - Event-based Simultaneous Localization and Mapping: A Comprehensive Survey [52.73728442921428]
Review of event-based vSLAM algorithms that exploit the benefits of asynchronous and irregular event streams for localization and mapping tasks.
Paper categorizes event-based vSLAM methods into four main categories: feature-based, direct, motion-compensation, and deep learning methods.
arXiv Detail & Related papers (2023-04-19T16:21:14Z) - Dual Memory Aggregation Network for Event-Based Object Detection with
Learnable Representation [79.02808071245634]
Event-based cameras are bio-inspired sensors that capture brightness change of every pixel in an asynchronous manner.
Event streams are divided into grids in the x-y-t coordinates for both positive and negative polarity, producing a set of pillars as 3D tensor representation.
Long memory is encoded in the hidden state of adaptive convLSTMs while short memory is modeled by computing spatial-temporal correlation between event pillars.
arXiv Detail & Related papers (2023-03-17T12:12:41Z) - Asynchronous Optimisation for Event-based Visual Odometry [53.59879499700895]
Event cameras open up new possibilities for robotic perception due to their low latency and high dynamic range.
We focus on event-based visual odometry (VO)
We propose an asynchronous structure-from-motion optimisation back-end.
arXiv Detail & Related papers (2022-03-02T11:28:47Z) - DEVO: Depth-Event Camera Visual Odometry in Challenging Conditions [30.892930944644853]
We present a novel real-time visual odometry framework for a stereo setup of a depth and high-resolution event camera.
Our framework balances accuracy and robustness against computational efficiency towards strong performance in challenging scenarios.
arXiv Detail & Related papers (2022-02-05T13:46:47Z) - 3D-FlowNet: Event-based optical flow estimation with 3D representation [2.062593640149623]
Event-based cameras can overpass frame-based cameras limitations for important tasks such as high-speed motion detection.
Deep Neural Networks are not well adapted to work with event data as they are asynchronous and discrete.
We propose 3D-FlowNet, a novel network architecture that can process the 3D input representation and output optical flow estimations.
arXiv Detail & Related papers (2022-01-28T17:28:15Z) - Improved Real-Time Monocular SLAM Using Semantic Segmentation on
Selective Frames [15.455647477995312]
monocular simultaneous localization and mapping (SLAM) is emerging in advanced driver assistance systems and autonomous driving.
This paper proposes an improved real-time monocular SLAM using deep learning-based semantic segmentation.
Experiments with six video sequences demonstrate that the proposed monocular SLAM system achieves significantly more accurate trajectory tracking accuracy.
arXiv Detail & Related papers (2021-04-30T22:34:45Z) - Monocular Quasi-Dense 3D Object Tracking [99.51683944057191]
A reliable and accurate 3D tracking framework is essential for predicting future locations of surrounding objects and planning the observer's actions in numerous applications such as autonomous driving.
We propose a framework that can effectively associate moving objects over time and estimate their full 3D bounding box information from a sequence of 2D images captured on a moving platform.
arXiv Detail & Related papers (2021-03-12T15:30:02Z) - Event-based Stereo Visual Odometry [42.77238738150496]
We present a solution to the problem of visual odometry from the data acquired by a stereo event-based camera rig.
We seek to maximize thetemporal consistency of stereo event-based data while using a simple and efficient representation.
arXiv Detail & Related papers (2020-07-30T15:53:28Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.