Learning Selective Sensor Fusion for States Estimation
- URL: http://arxiv.org/abs/1912.13077v2
- Date: Wed, 18 May 2022 10:42:16 GMT
- Title: Learning Selective Sensor Fusion for States Estimation
- Authors: Changhao Chen, Stefano Rosa, Chris Xiaoxuan Lu, Bing Wang, Niki
Trigoni, Andrew Markham
- Abstract summary: We propose SelectFusion, an end-to-end selective sensor fusion module.
During prediction, the network is able to assess the reliability of the latent features from different sensor modalities.
We extensively evaluate all fusion strategies in both public datasets and on progressively degraded datasets.
- Score: 47.76590539558037
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Autonomous vehicles and mobile robotic systems are typically equipped with
multiple sensors to provide redundancy. By integrating the observations from
different sensors, these mobile agents are able to perceive the environment and
estimate system states, e.g. locations and orientations. Although deep learning
approaches for multimodal odometry estimation and localization have gained
traction, they rarely focus on the issue of robust sensor fusion - a necessary
consideration to deal with noisy or incomplete sensor observations in the real
world. Moreover, current deep odometry models suffer from a lack of
interpretability. To this extent, we propose SelectFusion, an end-to-end
selective sensor fusion module which can be applied to useful pairs of sensor
modalities such as monocular images and inertial measurements, depth images and
LIDAR point clouds. Our model is a uniform framework that is not restricted to
specific modality or task. During prediction, the network is able to assess the
reliability of the latent features from different sensor modalities and
estimate trajectory both at scale and global pose. In particular, we propose
two fusion modules - a deterministic soft fusion and a stochastic hard fusion,
and offer a comprehensive study of the new strategies compared to trivial
direct fusion. We extensively evaluate all fusion strategies in both public
datasets and on progressively degraded datasets that present synthetic
occlusions, noisy and missing data and time misalignment between sensors, and
we investigate the effectiveness of the different fusion strategies in
attending the most reliable features, which in itself, provides insights into
the operation of the various models.
Related papers
- Guided Masked Self-Distillation Modeling for Distributed Multimedia Sensor Event Analysis [27.780058825326012]
We propose Guided Masked sELf-Distillation modeling (Guided-MELD) for inter-sensor relationship modeling.
Guided-MELD is expected to enable the system to effectively distill the fragmented or redundant target event information.
arXiv Detail & Related papers (2024-04-12T06:23:48Z) - Efficient Multi-Resolution Fusion for Remote Sensing Data with Label
Uncertainty [0.7832189413179361]
This paper presents a new method for fusing multi-modal and multi-resolution remote sensor data without requiring pixel-level training labels.
We propose a new method based on binary fuzzy measures, which reduces the search space and significantly improves the efficiency of the MIMRF framework.
arXiv Detail & Related papers (2024-02-07T17:34:32Z) - Log-Likelihood Score Level Fusion for Improved Cross-Sensor Smartphone
Periocular Recognition [52.15994166413364]
We employ fusion of several comparators to improve periocular performance when images from different smartphones are compared.
We use a probabilistic fusion framework based on linear logistic regression, in which fused scores tend to be log-likelihood ratios.
Our framework also provides an elegant and simple solution to handle signals from different devices, since same-sensor and cross-sensor score distributions are aligned and mapped to a common probabilistic domain.
arXiv Detail & Related papers (2023-11-02T13:43:44Z) - AFT-VO: Asynchronous Fusion Transformers for Multi-View Visual Odometry
Estimation [39.351088248776435]
We propose AFT-VO, a novel transformer-based sensor fusion architecture to estimate VO from multiple sensors.
Our framework combines predictions from asynchronous multi-view cameras and accounts for the time discrepancies of measurements coming from different sources.
Our experiments demonstrate that multi-view fusion for VO estimation provides robust and accurate trajectories, outperforming the state of the art in both challenging weather and lighting conditions.
arXiv Detail & Related papers (2022-06-26T19:29:08Z) - Learning Online Multi-Sensor Depth Fusion [100.84519175539378]
SenFuNet is a depth fusion approach that learns sensor-specific noise and outlier statistics.
We conduct experiments with various sensor combinations on the real-world CoRBS and Scene3D datasets.
arXiv Detail & Related papers (2022-04-07T10:45:32Z) - Target-aware Dual Adversarial Learning and a Multi-scenario
Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection [65.30079184700755]
This study addresses the issue of fusing infrared and visible images that appear differently for object detection.
Previous approaches discover commons underlying the two modalities and fuse upon the common space either by iterative optimization or deep networks.
This paper proposes a bilevel optimization formulation for the joint problem of fusion and detection, and then unrolls to a target-aware Dual Adversarial Learning (TarDAL) network for fusion and a commonly used detection network.
arXiv Detail & Related papers (2022-03-30T11:44:56Z) - Perception-aware Multi-sensor Fusion for 3D LiDAR Semantic Segmentation [59.42262859654698]
3D semantic segmentation is important in scene understanding for many applications, such as auto-driving and robotics.
Existing fusion-based methods may not achieve promising performance due to vast difference between two modalities.
In this work, we investigate a collaborative fusion scheme called perception-aware multi-sensor fusion (PMF) to exploit perceptual information from two modalities.
arXiv Detail & Related papers (2021-06-21T10:47:26Z) - GEM: Glare or Gloom, I Can Still See You -- End-to-End Multimodal Object
Detector [11.161639542268015]
We propose sensor-aware multi-modal fusion strategies for 2D object detection in harsh-lighting conditions.
Our network learns to estimate the measurement reliability of each sensor modality in the form of scalar weights and masks.
We show that the proposed strategies out-perform the existing state-of-the-art methods on the FLIR-Thermal dataset.
arXiv Detail & Related papers (2021-02-24T14:56:37Z) - Multi-Objective Bayesian Optimisation and Joint Inversion for Active
Sensor Fusion [22.04258832800079]
We propose a framework for multi-objective optimisation and inverse problems given an expensive cost function for allocating new measurements.
This new method is devised to jointly solve multi-linear forward models of 2D-sensor data and 3D-geophysical properties.
We demonstrate the advantages on a specific example of a joint inverse problem, recommending where to place new drill-core measurements given 2D gravity and magnetic sensor data.
arXiv Detail & Related papers (2020-10-12T01:23:41Z) - Deep Soft Procrustes for Markerless Volumetric Sensor Alignment [81.13055566952221]
In this work, we improve markerless data-driven correspondence estimation to achieve more robust multi-sensor spatial alignment.
We incorporate geometric constraints in an end-to-end manner into a typical segmentation based model and bridge the intermediate dense classification task with the targeted pose estimation one.
Our model is experimentally shown to achieve similar results with marker-based methods and outperform the markerless ones, while also being robust to the pose variations of the calibration structure.
arXiv Detail & Related papers (2020-03-23T10:51:32Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.