RoboBEV: Towards Robust Bird's Eye View Perception under Corruptions

• URL: http://arxiv.org/abs/2304.06719v1
• Date: Thu, 13 Apr 2023 17:59:46 GMT
• Title: RoboBEV: Towards Robust Bird's Eye View Perception under Corruptions
• Authors: Shaoyuan Xie, Lingdong Kong, Wenwei Zhang, Jiawei Ren, Liang Pan, Kai Chen, Ziwei Liu
• Abstract summary: We introduce RoboBEV, a comprehensive benchmark suite that encompasses eight distinct corruptions, including Bright, Dark, Fog, Snow, Motion Blur, Color Quant, Camera Crash, and Frame Lost. Based on it, we undertake extensive evaluations across a wide range of BEV-based models to understand their resilience and reliability. Our findings provide valuable insights for designing future BEV models that can achieve both accuracy and robustness in real-world deployments.
• Score: 34.111443808494506
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: The recent advances in camera-based bird's eye view (BEV) representation exhibit great potential for in-vehicle 3D perception. Despite the substantial progress achieved on standard benchmarks, the robustness of BEV algorithms has not been thoroughly examined, which is critical for safe operations. To bridge this gap, we introduce RoboBEV, a comprehensive benchmark suite that encompasses eight distinct corruptions, including Bright, Dark, Fog, Snow, Motion Blur, Color Quant, Camera Crash, and Frame Lost. Based on it, we undertake extensive evaluations across a wide range of BEV-based models to understand their resilience and reliability. Our findings indicate a strong correlation between absolute performance on in-distribution and out-of-distribution datasets. Nonetheless, there are considerable variations in relative performance across different approaches. Our experiments further demonstrate that pre-training and depth-free BEV transformation has the potential to enhance out-of-distribution robustness. Additionally, utilizing long and rich temporal information largely helps with robustness. Our findings provide valuable insights for designing future BEV models that can achieve both accuracy and robustness in real-world deployments.

Related papers

• Robust Bird's Eye View Segmentation by Adapting DINOv2 [3.236198583140341]
  We adapt a vision foundational model, DINOv2, to BEV estimation using Low Rank Adaptation (LoRA) Our experiments show increased robustness of BEV perception under various corruptions. We also showcase the effectiveness of the adapted representations in terms of fewer learnable parameters and faster convergence during training.
  arXiv  Detail & Related papers  (2024-09-16T12:23:35Z)
• Benchmarking and Improving Bird's Eye View Perception Robustness in Autonomous Driving [55.93813178692077]
  We present RoboBEV, an extensive benchmark suite designed to evaluate the resilience of BEV algorithms. We assess 33 state-of-the-art BEV-based perception models spanning tasks like detection, map segmentation, depth estimation, and occupancy prediction. Our experimental results also underline the efficacy of strategies like pre-training and depth-free BEV transformations in enhancing robustness against out-of-distribution data.
  arXiv  Detail & Related papers  (2024-05-27T17:59:39Z)
• CLIP-BEVFormer: Enhancing Multi-View Image-Based BEV Detector with Ground Truth Flow [20.550935390111686]
  We introduce CLIP-BEVFormer, a novel approach to enhance the multi-view image-derived BEV backbones with ground truth information flow. We conduct extensive experiments on the challenging nuScenes dataset and showcase significant and consistent improvements over the SOTA.
  arXiv  Detail & Related papers  (2024-03-13T19:21:03Z)
• DA-BEV: Unsupervised Domain Adaptation for Bird's Eye View Perception [104.87876441265593]
  Camera-only Bird's Eye View (BEV) has demonstrated great potential in environment perception in a 3D space. Unsupervised domain adaptive BEV, which effective learning from various unlabelled target data, is far under-explored. We design DA-BEV, the first domain adaptive camera-only BEV framework that addresses domain adaptive BEV challenges by exploiting the complementary nature of image-view features and BEV features.
  arXiv  Detail & Related papers  (2024-01-13T04:21:24Z)
• Instance-aware Multi-Camera 3D Object Detection with Structural Priors Mining and Self-Boosting Learning [93.71280187657831]
  Camera-based bird-eye-view (BEV) perception paradigm has made significant progress in the autonomous driving field. We propose IA-BEV, which integrates image-plane instance awareness into the depth estimation process within a BEV-based detector.
  arXiv  Detail & Related papers  (2023-12-13T09:24:42Z)
• BroadBEV: Collaborative LiDAR-camera Fusion for Broad-sighted Bird's Eye View Map Construction [31.664613321775516]
  We propose a broad BEV fusion (BroadBEV) that addresses the problems with a spatial synchronization approach of cross-modality. Our strategy aims to enhance camera BEV estimation for a broad-sighted perception while simultaneously improving the completion of LiDAR's sparsity in the entire BEV space.
  arXiv  Detail & Related papers  (2023-09-20T07:55:57Z)
• Robo3D: Towards Robust and Reliable 3D Perception against Corruptions [58.306694836881235]
  We present Robo3D, the first comprehensive benchmark heading toward probing the robustness of 3D detectors and segmentors under out-of-distribution scenarios. We consider eight corruption types stemming from severe weather conditions, external disturbances, and internal sensor failure. We propose a density-insensitive training framework along with a simple flexible voxelization strategy to enhance the model resiliency.
  arXiv  Detail & Related papers  (2023-03-30T17:59:17Z)
• Understanding the Robustness of 3D Object Detection with Bird's-Eye-View Representations in Autonomous Driving [31.98600806479808]
  Bird's-Eye-View (BEV) representations have significantly improved the performance of 3D detectors with camera inputs on popular benchmarks. We evaluate the natural and adversarial robustness of various representative models under extensive settings. We propose a 3D consistent patch attack by applying adversarial patches in thetemporal 3D space to guarantee the consistency.
  arXiv  Detail & Related papers  (2023-03-30T11:16:58Z)
• Delving into the Devils of Bird's-eye-view Perception: A Review, Evaluation and Recipe [115.31507979199564]
  Learning powerful representations in bird's-eye-view (BEV) for perception tasks is trending and drawing extensive attention both from industry and academia. As sensor configurations get more complex, integrating multi-source information from different sensors and representing features in a unified view come of vital importance. The core problems for BEV perception lie in (a) how to reconstruct the lost 3D information via view transformation from perspective view to BEV; (b) how to acquire ground truth annotations in BEV grid; and (d) how to adapt and generalize algorithms as sensor configurations vary across different scenarios.
  arXiv  Detail & Related papers  (2022-09-12T15:29:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.