On Offline Evaluation of 3D Object Detection for Autonomous Driving

• URL: http://arxiv.org/abs/2308.12779v1
• Date: Thu, 24 Aug 2023 13:31:51 GMT
• Title: On Offline Evaluation of 3D Object Detection for Autonomous Driving
• Authors: Tim Schreier, Katrin Renz, Andreas Geiger, Kashyap Chitta
• Abstract summary: We measure how predictive different detection metrics are of driving performance when detectors are integrated into a full self-driving stack. We find that the nuScenes Detection Score has a higher correlation to driving performance than the widely used average precision metric.
• Score: 33.16617625256519
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Prior work in 3D object detection evaluates models using offline metrics like average precision since closed-loop online evaluation on the downstream driving task is costly. However, it is unclear how indicative offline results are of driving performance. In this work, we perform the first empirical evaluation measuring how predictive different detection metrics are of driving performance when detectors are integrated into a full self-driving stack. We conduct extensive experiments on urban driving in the CARLA simulator using 16 object detection models. We find that the nuScenes Detection Score has a higher correlation to driving performance than the widely used average precision metric. In addition, our results call for caution on the exclusive reliance on the emerging class of `planner-centric' metrics.

Related papers

• Automatic Odometry-Less OpenDRIVE Generation From Sparse Point Clouds [1.3351610617039973]
  High-resolution road representations are a key factor for the success of automated driving functions. This paper proposes a novel approach to generate realistic road representations based solely on point cloud information.
  arXiv  Detail & Related papers  (2024-05-13T08:26:24Z)
• Guiding Attention in End-to-End Driving Models [49.762868784033785]
  Vision-based end-to-end driving models trained by imitation learning can lead to affordable solutions for autonomous driving. We study how to guide the attention of these models to improve their driving quality by adding a loss term during training. In contrast to previous work, our method does not require these salient semantic maps to be available during testing time.
  arXiv  Detail & Related papers  (2024-04-30T23:18:51Z)
• Unsupervised Domain Adaptation for Self-Driving from Past Traversal Features [69.47588461101925]
  We propose a method to adapt 3D object detectors to new driving environments. Our approach enhances LiDAR-based detection models using spatial quantized historical features. Experiments on real-world datasets demonstrate significant improvements.
  arXiv  Detail & Related papers  (2023-09-21T15:00:31Z)
• Improving Online Lane Graph Extraction by Object-Lane Clustering [106.71926896061686]
  We propose an architecture and loss formulation to improve the accuracy of local lane graph estimates. The proposed method learns to assign the objects to centerlines by considering the centerlines as cluster centers. We show that our method can achieve significant performance improvements by using the outputs of existing 3D object detection methods.
  arXiv  Detail & Related papers  (2023-07-20T15:21:28Z)
• Towards Driving-Oriented Metric for Lane Detection Models [19.81163190104571]
  We design 2 new driving-oriented metrics for lane detection: End-to-End Lateral Deviation metric (E2E-LD) and Per-frame Simulated Lateral Deviation metric (PSLD) To evaluate the validity of the metrics, we conduct a large-scale empirical study with 4 major types of lane detection approaches on the TuSimple dataset and our newly constructed dataset Comma2k19-LD.
  arXiv  Detail & Related papers  (2022-03-31T07:24:44Z)
• One Million Scenes for Autonomous Driving: ONCE Dataset [91.94189514073354]
  We introduce the ONCE dataset for 3D object detection in the autonomous driving scenario. The data is selected from 144 driving hours, which is 20x longer than the largest 3D autonomous driving dataset available. We reproduce and evaluate a variety of self-supervised and semi-supervised methods on the ONCE dataset.
  arXiv  Detail & Related papers  (2021-06-21T12:28:08Z)
• Deep Evaluation Metric: Learning to Evaluate Simulated Radar Point Clouds for Virtual Testing of Autonomous Driving [0.0]
  The usage of environment sensor models for virtual testing is a promising approach to reduce the testing effort of autonomous driving. In this work, we train a neural network to distinguish real and simulated radar sensor data. We propose the classifier's confidence score for the real radar point cloud' class as a metric to determine the degree of fidelity of synthetically generated radar data.
  arXiv  Detail & Related papers  (2021-04-14T11:04:50Z)
• IntentNet: Learning to Predict Intention from Raw Sensor Data [86.74403297781039]
  In this paper, we develop a one-stage detector and forecaster that exploits both 3D point clouds produced by a LiDAR sensor as well as dynamic maps of the environment. Our multi-task model achieves better accuracy than the respective separate modules while saving computation, which is critical to reducing reaction time in self-driving applications.
  arXiv  Detail & Related papers  (2021-01-20T00:31:52Z)
• Beelines: Motion Prediction Metrics for Self-Driving Safety and Comfort [4.670814682436471]
  We propose two complementary metrics that quantify the effects of motion prediction on safety. Our safety metric has a significantly better signal-to-noise ratio than displacement error in identifying unsafe events.
  arXiv  Detail & Related papers  (2020-11-01T01:09:23Z)
• Learning to Evaluate Perception Models Using Planner-Centric Metrics [104.33349410009161]
  We propose a principled metric for 3D object detection specifically for the task of self-driving. We find that our metric penalizes many of the mistakes that other metrics penalize by design. For human evaluation, we generate scenes in which standard metrics and our metric disagree and find that humans side with our metric 79% of the time.
  arXiv  Detail & Related papers  (2020-04-19T02:14:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.