Robust sensor fusion against on-vehicle sensor staleness

• URL: http://arxiv.org/abs/2506.05780v1
• Date: Fri, 06 Jun 2025 06:18:54 GMT
• Title: Robust sensor fusion against on-vehicle sensor staleness
• Authors: Meng Fan, Yifan Zuo, Patrick Blaes, Harley Montgomery, Subhasis Das,
• Abstract summary: Temporal misalignment between sensor modalities leads to inconsistent object state estimates.<n>We present a novel and model-agnostic approach to address this problem.<n>Our approach reaches consistently good performance across both synchronized and stale conditions.
• Score: 5.473556120706307
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Sensor fusion is crucial for a performant and robust Perception system in autonomous vehicles, but sensor staleness, where data from different sensors arrives with varying delays, poses significant challenges. Temporal misalignment between sensor modalities leads to inconsistent object state estimates, severely degrading the quality of trajectory predictions that are critical for safety. We present a novel and model-agnostic approach to address this problem via (1) a per-point timestamp offset feature (for LiDAR and radar both relative to camera) that enables fine-grained temporal awareness in sensor fusion, and (2) a data augmentation strategy that simulates realistic sensor staleness patterns observed in deployed vehicles. Our method is integrated into a perspective-view detection model that consumes sensor data from multiple LiDARs, radars and cameras. We demonstrate that while a conventional model shows significant regressions when one sensor modality is stale, our approach reaches consistently good performance across both synchronized and stale conditions.

Related papers

• MSC-Bench: Benchmarking and Analyzing Multi-Sensor Corruption for Driving Perception [9.575044300747061]
  Multi-sensor fusion models play a crucial role in autonomous driving perception, particularly in tasks like 3D object detection and HD map construction.<n>These models provide essential and comprehensive static environmental information for autonomous driving systems.<n>While camera-LiDAR fusion methods have shown promising results, they often depend on complete sensor inputs.<n>This reliance can lead to low robustness and potential failures when sensors are corrupted or missing, raising significant safety concerns.<n>To tackle this challenge, we introduce the Multi-Sensor Corruption Benchmark (MSC-Bench), the first comprehensive benchmark aimed at evaluating the robustness of multi-sensor autonomous driving perception models against various sensor corruption
  arXiv  Detail & Related papers  (2025-01-02T03:38:46Z)
• CAFuser: Condition-Aware Multimodal Fusion for Robust Semantic Perception of Driving Scenes [56.52618054240197]
  We propose a novel, condition-aware multimodal fusion approach for robust semantic perception of driving scenes.<n>Our method, CAFuser, uses an RGB camera input to classify environmental conditions and generate a Condition Token.<n>Our model significantly improves robustness and accuracy, especially in adverse-condition scenarios.
  arXiv  Detail & Related papers  (2024-10-14T17:56:20Z)
• Increasing the Robustness of Model Predictions to Missing Sensors in Earth Observation [5.143097874851516]
  We study two novel methods tailored for multi-sensor scenarios, namely Input Sensor Dropout (ISensD) and Ensemble Sensor Invariant (ESensI) We demonstrate that these methods effectively increase the robustness of model predictions to missing sensors. We observe that ensemble multi-sensor models are the most robust to the lack of sensors.
  arXiv  Detail & Related papers  (2024-07-22T09:58:29Z)
• OOSTraj: Out-of-Sight Trajectory Prediction With Vision-Positioning Denoising [49.86409475232849]
  Trajectory prediction is fundamental in computer vision and autonomous driving. Existing approaches in this field often assume precise and complete observational data. We present a novel method for out-of-sight trajectory prediction that leverages a vision-positioning technique.
  arXiv  Detail & Related papers  (2024-04-02T18:30:29Z)
• Extrinsic Camera Calibration with Semantic Segmentation [60.330549990863624]
  We present an extrinsic camera calibration approach that automatizes the parameter estimation by utilizing semantic segmentation information. Our approach relies on a coarse initial measurement of the camera pose and builds on lidar sensors mounted on a vehicle. We evaluate our method on simulated and real-world data to demonstrate low error measurements in the calibration results.
  arXiv  Detail & Related papers  (2022-08-08T07:25:03Z)
• Sensor Adversarial Traits: Analyzing Robustness of 3D Object Detection Sensor Fusion Models [16.823829387723524]
  We analyze the robustness of a high-performance, open source sensor fusion model architecture towards adversarial attacks. We find that despite the use of a LIDAR sensor, the model is vulnerable to our purposefully crafted image-based adversarial attacks.
  arXiv  Detail & Related papers  (2021-09-13T23:38:42Z)
• Efficient and Robust LiDAR-Based End-to-End Navigation [132.52661670308606]
  We present an efficient and robust LiDAR-based end-to-end navigation framework. We propose Fast-LiDARNet that is based on sparse convolution kernel optimization and hardware-aware model design. We then propose Hybrid Evidential Fusion that directly estimates the uncertainty of the prediction from only a single forward pass.
  arXiv  Detail & Related papers  (2021-05-20T17:52:37Z)
• Complex-valued Convolutional Neural Networks for Enhanced Radar Signal Denoising and Interference Mitigation [73.0103413636673]
  We propose the use of Complex-Valued Convolutional Neural Networks (CVCNNs) to address the issue of mutual interference between radar sensors. CVCNNs increase data efficiency, speeds up network training and substantially improves the conservation of phase information during interference removal.
  arXiv  Detail & Related papers  (2021-04-29T10:06:29Z)
• On the Role of Sensor Fusion for Object Detection in Future Vehicular Networks [25.838878314196375]
  We evaluate how using a combination of different sensors affects the detection of the environment in which the vehicles move and operate. The final objective is to identify the optimal setup that would minimize the amount of data to be distributed over the channel.
  arXiv  Detail & Related papers  (2021-04-23T18:58:37Z)
• Investigating the Effect of Sensor Modalities in Multi-Sensor Detection-Prediction Models [8.354898936252516]
  We focus on the contribution of sensor modalities towards the model performance. In addition, we investigate the use of sensor dropout to mitigate the above-mentioned issues.
  arXiv  Detail & Related papers  (2021-01-09T03:21:36Z)
• Towards robust sensing for Autonomous Vehicles: An adversarial perspective [82.83630604517249]
  It is of primary importance that the resulting decisions are robust to perturbations. Adversarial perturbations are purposefully crafted alterations of the environment or of the sensory measurements. A careful evaluation of the vulnerabilities of their sensing system(s) is necessary in order to build and deploy safer systems.
  arXiv  Detail & Related papers  (2020-07-14T05:25:15Z)
• Learning Selective Sensor Fusion for States Estimation [47.76590539558037]
  We propose SelectFusion, an end-to-end selective sensor fusion module. During prediction, the network is able to assess the reliability of the latent features from different sensor modalities. We extensively evaluate all fusion strategies in both public datasets and on progressively degraded datasets.
  arXiv  Detail & Related papers  (2019-12-30T20:25:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.