Learning Selective Sensor Fusion for States Estimation
- URL: http://arxiv.org/abs/1912.13077v2
- Date: Wed, 18 May 2022 10:42:16 GMT
- Title: Learning Selective Sensor Fusion for States Estimation
- Authors: Changhao Chen, Stefano Rosa, Chris Xiaoxuan Lu, Bing Wang, Niki
Trigoni, Andrew Markham
- Abstract summary: We propose SelectFusion, an end-to-end selective sensor fusion module.
During prediction, the network is able to assess the reliability of the latent features from different sensor modalities.
We extensively evaluate all fusion strategies in both public datasets and on progressively degraded datasets.
- Score: 47.76590539558037
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Autonomous vehicles and mobile robotic systems are typically equipped with
multiple sensors to provide redundancy. By integrating the observations from
different sensors, these mobile agents are able to perceive the environment and
estimate system states, e.g. locations and orientations. Although deep learning
approaches for multimodal odometry estimation and localization have gained
traction, they rarely focus on the issue of robust sensor fusion - a necessary
consideration to deal with noisy or incomplete sensor observations in the real
world. Moreover, current deep odometry models suffer from a lack of
interpretability. To this extent, we propose SelectFusion, an end-to-end
selective sensor fusion module which can be applied to useful pairs of sensor
modalities such as monocular images and inertial measurements, depth images and
LIDAR point clouds. Our model is a uniform framework that is not restricted to
specific modality or task. During prediction, the network is able to assess the
reliability of the latent features from different sensor modalities and
estimate trajectory both at scale and global pose. In particular, we propose
two fusion modules - a deterministic soft fusion and a stochastic hard fusion,
and offer a comprehensive study of the new strategies compared to trivial
direct fusion. We extensively evaluate all fusion strategies in both public
datasets and on progressively degraded datasets that present synthetic
occlusions, noisy and missing data and time misalignment between sensors, and
we investigate the effectiveness of the different fusion strategies in
attending the most reliable features, which in itself, provides insights into
the operation of the various models.
Related papers
- Graph-Based Multi-Modal Sensor Fusion for Autonomous Driving [3.770103075126785]
We introduce a novel approach to multi-modal sensor fusion, focusing on developing a graph-based state representation.
We present a Sensor-Agnostic Graph-Aware Kalman Filter, the first online state estimation technique designed to fuse multi-modal graphs.
We validate the effectiveness of our proposed framework through extensive experiments conducted on both synthetic and real-world driving datasets.
arXiv Detail & Related papers (2024-11-06T06:58:17Z) - Condition-Aware Multimodal Fusion for Robust Semantic Perception of Driving Scenes [56.52618054240197]
We propose a novel, condition-aware multimodal fusion approach for robust semantic perception of driving scenes.
Our method, CAFuser, uses an RGB camera input to classify environmental conditions and generate a Condition Token that guides the fusion of multiple sensor modalities.
We set the new state of the art with CAFuser on the MUSES dataset with 59.7 PQ for multimodal panoptic segmentation and 78.2 mIoU for semantic segmentation, ranking first on the public benchmarks.
arXiv Detail & Related papers (2024-10-14T17:56:20Z) - Guided Masked Self-Distillation Modeling for Distributed Multimedia Sensor Event Analysis [27.780058825326012]
We propose Guided Masked sELf-Distillation modeling (Guided-MELD) for inter-sensor relationship modeling.
Guided-MELD is expected to enable the system to effectively distill the fragmented or redundant target event information.
arXiv Detail & Related papers (2024-04-12T06:23:48Z) - Efficient Multi-Resolution Fusion for Remote Sensing Data with Label
Uncertainty [0.7832189413179361]
This paper presents a new method for fusing multi-modal and multi-resolution remote sensor data without requiring pixel-level training labels.
We propose a new method based on binary fuzzy measures, which reduces the search space and significantly improves the efficiency of the MIMRF framework.
arXiv Detail & Related papers (2024-02-07T17:34:32Z) - Log-Likelihood Score Level Fusion for Improved Cross-Sensor Smartphone
Periocular Recognition [52.15994166413364]
We employ fusion of several comparators to improve periocular performance when images from different smartphones are compared.
We use a probabilistic fusion framework based on linear logistic regression, in which fused scores tend to be log-likelihood ratios.
Our framework also provides an elegant and simple solution to handle signals from different devices, since same-sensor and cross-sensor score distributions are aligned and mapped to a common probabilistic domain.
arXiv Detail & Related papers (2023-11-02T13:43:44Z) - Learning Online Multi-Sensor Depth Fusion [100.84519175539378]
SenFuNet is a depth fusion approach that learns sensor-specific noise and outlier statistics.
We conduct experiments with various sensor combinations on the real-world CoRBS and Scene3D datasets.
arXiv Detail & Related papers (2022-04-07T10:45:32Z) - Target-aware Dual Adversarial Learning and a Multi-scenario
Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection [65.30079184700755]
This study addresses the issue of fusing infrared and visible images that appear differently for object detection.
Previous approaches discover commons underlying the two modalities and fuse upon the common space either by iterative optimization or deep networks.
This paper proposes a bilevel optimization formulation for the joint problem of fusion and detection, and then unrolls to a target-aware Dual Adversarial Learning (TarDAL) network for fusion and a commonly used detection network.
arXiv Detail & Related papers (2022-03-30T11:44:56Z) - GEM: Glare or Gloom, I Can Still See You -- End-to-End Multimodal Object
Detector [11.161639542268015]
We propose sensor-aware multi-modal fusion strategies for 2D object detection in harsh-lighting conditions.
Our network learns to estimate the measurement reliability of each sensor modality in the form of scalar weights and masks.
We show that the proposed strategies out-perform the existing state-of-the-art methods on the FLIR-Thermal dataset.
arXiv Detail & Related papers (2021-02-24T14:56:37Z) - Multi-Objective Bayesian Optimisation and Joint Inversion for Active
Sensor Fusion [22.04258832800079]
We propose a framework for multi-objective optimisation and inverse problems given an expensive cost function for allocating new measurements.
This new method is devised to jointly solve multi-linear forward models of 2D-sensor data and 3D-geophysical properties.
We demonstrate the advantages on a specific example of a joint inverse problem, recommending where to place new drill-core measurements given 2D gravity and magnetic sensor data.
arXiv Detail & Related papers (2020-10-12T01:23:41Z) - Deep Soft Procrustes for Markerless Volumetric Sensor Alignment [81.13055566952221]
In this work, we improve markerless data-driven correspondence estimation to achieve more robust multi-sensor spatial alignment.
We incorporate geometric constraints in an end-to-end manner into a typical segmentation based model and bridge the intermediate dense classification task with the targeted pose estimation one.
Our model is experimentally shown to achieve similar results with marker-based methods and outperform the markerless ones, while also being robust to the pose variations of the calibration structure.
arXiv Detail & Related papers (2020-03-23T10:51:32Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.