MultiCorrupt: A Multi-Modal Robustness Dataset and Benchmark of LiDAR-Camera Fusion for 3D Object Detection

• URL: http://arxiv.org/abs/2402.11677v3
• Date: Sat, 20 Apr 2024 13:00:25 GMT
• Title: MultiCorrupt: A Multi-Modal Robustness Dataset and Benchmark of LiDAR-Camera Fusion for 3D Object Detection
• Authors: Till Beemelmanns, Quan Zhang, Christian Geller, Lutz Eckstein,
• Abstract summary: Multi-modal 3D object detection models for automated driving have demonstrated exceptional performance on computer vision benchmarks like nuScenes. However, their reliance on densely sampled LiDAR point clouds and meticulously calibrated sensor arrays poses challenges for real-world applications. We introduce MultiCorrupt, a benchmark designed to evaluate the robustness of multi-modal 3D object detectors against ten distinct types of corruptions.
• Score: 5.462358595564476
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multi-modal 3D object detection models for automated driving have demonstrated exceptional performance on computer vision benchmarks like nuScenes. However, their reliance on densely sampled LiDAR point clouds and meticulously calibrated sensor arrays poses challenges for real-world applications. Issues such as sensor misalignment, miscalibration, and disparate sampling frequencies lead to spatial and temporal misalignment in data from LiDAR and cameras. Additionally, the integrity of LiDAR and camera data is often compromised by adverse environmental conditions such as inclement weather, leading to occlusions and noise interference. To address this challenge, we introduce MultiCorrupt, a comprehensive benchmark designed to evaluate the robustness of multi-modal 3D object detectors against ten distinct types of corruptions. We evaluate five state-of-the-art multi-modal detectors on MultiCorrupt and analyze their performance in terms of their resistance ability. Our results show that existing methods exhibit varying degrees of robustness depending on the type of corruption and their fusion strategy. We provide insights into which multi-modal design choices make such models robust against certain perturbations. The dataset generation code and benchmark are open-sourced at https://github.com/ika-rwth-aachen/MultiCorrupt.

Related papers

• Robust Multimodal 3D Object Detection via Modality-Agnostic Decoding and Proximity-based Modality Ensemble [15.173314907900842]
  Existing 3D object detection methods rely heavily on the LiDAR sensor. We propose MEFormer to address the LiDAR over-reliance problem. Our MEFormer achieves state-of-the-art performance of 73.9% NDS and 71.5% mAP in the nuScenes validation set.
  arXiv  Detail & Related papers  (2024-07-27T03:21:44Z)
• M3DM-NR: RGB-3D Noisy-Resistant Industrial Anomaly Detection via Multimodal Denoising [63.39134873744748]
  Existing industrial anomaly detection methods primarily concentrate on unsupervised learning with pristine RGB images. This paper proposes a novel noise-resistant M3DM-NR framework to leverage strong multi-modal discriminative capabilities of CLIP. Extensive experiments show that M3DM-NR outperforms state-of-the-art methods in 3D-RGB multi-modal noisy anomaly detection.
  arXiv  Detail & Related papers  (2024-06-04T12:33:02Z)
• Towards Unified 3D Object Detection via Algorithm and Data Unification [70.27631528933482]
  We build the first unified multi-modal 3D object detection benchmark MM- Omni3D and extend the aforementioned monocular detector to its multi-modal version. We name the designed monocular and multi-modal detectors as UniMODE and MM-UniMODE, respectively.
  arXiv  Detail & Related papers  (2024-02-28T18:59:31Z)
• Towards a Robust Sensor Fusion Step for 3D Object Detection on Corrupted Data [4.3012765978447565]
  This work presents a novel fusion step that addresses data corruptions and makes sensor fusion for 3D object detection more robust. We demonstrate that our method performs on par with state-of-the-art approaches on normal data and outperforms them on misaligned data.
  arXiv  Detail & Related papers  (2023-06-12T18:06:29Z)
• Multi-Modal 3D Object Detection by Box Matching [109.43430123791684]
  We propose a novel Fusion network by Box Matching (FBMNet) for multi-modal 3D detection. With the learned assignments between 3D and 2D object proposals, the fusion for detection can be effectively performed by combing their ROI features.
  arXiv  Detail & Related papers  (2023-05-12T18:08:51Z)
• Benchmarking Robustness of 3D Object Detection to Common Corruptions in Autonomous Driving [44.753797839280516]
  Existing 3D detectors lack robustness to real-world corruptions caused by adverse weathers, sensor noises, etc. We benchmark 27 types of common corruptions for both LiDAR and camera inputs considering real-world driving scenarios. We conduct large-scale experiments on 24 diverse 3D object detection models to evaluate their robustness.
  arXiv  Detail & Related papers  (2023-03-20T11:45:54Z)
• Multimodal Industrial Anomaly Detection via Hybrid Fusion [59.16333340582885]
  We propose a novel multimodal anomaly detection method with hybrid fusion scheme. Our model outperforms the state-of-the-art (SOTA) methods on both detection and segmentation precision on MVTecD-3 AD dataset.
  arXiv  Detail & Related papers  (2023-03-01T15:48:27Z)
• Benchmarking the Robustness of LiDAR-Camera Fusion for 3D Object Detection [58.81316192862618]
  Two critical sensors for 3D perception in autonomous driving are the camera and the LiDAR. fusing these two modalities can significantly boost the performance of 3D perception models. We benchmark the state-of-the-art fusion methods for the first time.
  arXiv  Detail & Related papers  (2022-05-30T09:35:37Z)
• EPNet++: Cascade Bi-directional Fusion for Multi-Modal 3D Object Detection [56.03081616213012]
  We propose EPNet++ for multi-modal 3D object detection by introducing a novel Cascade Bi-directional Fusion(CB-Fusion) module. The proposed CB-Fusion module boosts the plentiful semantic information of point features with the image features in a cascade bi-directional interaction fusion manner. The experiment results on the KITTI, JRDB and SUN-RGBD datasets demonstrate the superiority of EPNet++ over the state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-12-21T10:48:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.