Learning to Evaluate Perception Models Using Planner-Centric Metrics

• URL: http://arxiv.org/abs/2004.08745v1
• Date: Sun, 19 Apr 2020 02:14:00 GMT
• Title: Learning to Evaluate Perception Models Using Planner-Centric Metrics
• Authors: Jonah Philion, Amlan Kar, Sanja Fidler
• Abstract summary: We propose a principled metric for 3D object detection specifically for the task of self-driving. We find that our metric penalizes many of the mistakes that other metrics penalize by design. For human evaluation, we generate scenes in which standard metrics and our metric disagree and find that humans side with our metric 79% of the time.
• Score: 104.33349410009161
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Variants of accuracy and precision are the gold-standard by which the computer vision community measures progress of perception algorithms. One reason for the ubiquity of these metrics is that they are largely task-agnostic; we in general seek to detect zero false negatives or positives. The downside of these metrics is that, at worst, they penalize all incorrect detections equally without conditioning on the task or scene, and at best, heuristics need to be chosen to ensure that different mistakes count differently. In this paper, we propose a principled metric for 3D object detection specifically for the task of self-driving. The core idea behind our metric is to isolate the task of object detection and measure the impact the produced detections would induce on the downstream task of driving. Without hand-designing it to, we find that our metric penalizes many of the mistakes that other metrics penalize by design. In addition, our metric downweighs detections based on additional factors such as distance from a detection to the ego car and the speed of the detection in intuitive ways that other detection metrics do not. For human evaluation, we generate scenes in which standard metrics and our metric disagree and find that humans side with our metric 79% of the time. Our project page including an evaluation server can be found at https://nv-tlabs.github.io/detection-relevance.

Related papers

• Uncertainty Estimation for 3D Object Detection via Evidential Learning [63.61283174146648]
  We introduce a framework for quantifying uncertainty in 3D object detection by leveraging an evidential learning loss on Bird's Eye View representations in the 3D detector. We demonstrate both the efficacy and importance of these uncertainty estimates on identifying out-of-distribution scenes, poorly localized objects, and missing (false negative) detections.
  arXiv  Detail & Related papers  (2024-10-31T13:13:32Z)
• Learning 3D Perception from Others' Predictions [64.09115694891679]
  We investigate a new scenario to construct 3D object detectors: learning from the predictions of a nearby unit that is equipped with an accurate detector. For example, when a self-driving car enters a new area, it may learn from other traffic participants whose detectors have been optimized for that area.
  arXiv  Detail & Related papers  (2024-10-03T16:31:28Z)
• Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuracies [24.26653413077486]
  Ten years ago a single metric, BLEU, governed progress in machine translation research. This paper investigates the "dynamic range" of modern metrics.
  arXiv  Detail & Related papers  (2024-01-12T18:47:40Z)
• On Offline Evaluation of 3D Object Detection for Autonomous Driving [33.16617625256519]
  We measure how predictive different detection metrics are of driving performance when detectors are integrated into a full self-driving stack. We find that the nuScenes Detection Score has a higher correlation to driving performance than the widely used average precision metric.
  arXiv  Detail & Related papers  (2023-08-24T13:31:51Z)
• The Glass Ceiling of Automatic Evaluation in Natural Language Generation [60.59732704936083]
  We take a step back and analyze recent progress by comparing the body of existing automatic metrics and human metrics. Our extensive statistical analysis reveals surprising findings: automatic metrics -- old and new -- are much more similar to each other than to humans.
  arXiv  Detail & Related papers  (2022-08-31T01:13:46Z)
• Exploring Credibility Scoring Metrics of Perception Systems for Autonomous Driving [0.0]
  We show that offline metrics can be used to account for real-world corruptions such as poor weather conditions. This is a clear next step as it can allow for error-free autonomous vehicle perception and safer time-critical and safety-critical decision-making.
  arXiv  Detail & Related papers  (2021-12-22T03:17:14Z)
• Injecting Planning-Awareness into Prediction and Detection Evaluation [42.228191984697006]
  We take a step back and critically assess current evaluation metrics, proposing task-aware metrics as a better measure of performance in systems where they are deployed. Experiments on an illustrative simulation as well as real-world autonomous driving data validate that our proposed task-aware metrics are able to account for outcome asymmetry and provide a better estimate of a model's closed-loop performance.
  arXiv  Detail & Related papers  (2021-10-07T08:52:48Z)
• CertainNet: Sampling-free Uncertainty Estimation for Object Detection [65.28989536741658]
  Estimating the uncertainty of a neural network plays a fundamental role in safety-critical settings. In this work, we propose a novel sampling-free uncertainty estimation method for object detection. We call it CertainNet, and it is the first to provide separate uncertainties for each output signal: objectness, class, location and size.
  arXiv  Detail & Related papers  (2021-10-04T17:59:31Z)
• Rethinking Trajectory Forecasting Evaluation [42.228191984697006]
  We take a step back and critically evaluate current trajectory forecasting metrics. We propose task-aware metrics as a better measure of performance in systems where prediction is being deployed.
  arXiv  Detail & Related papers  (2021-07-21T18:20:03Z)
• Provably Robust Metric Learning [98.50580215125142]
  We show that existing metric learning algorithms can result in metrics that are less robust than the Euclidean distance. We propose a novel metric learning algorithm to find a Mahalanobis distance that is robust against adversarial perturbations. Experimental results show that the proposed metric learning algorithm improves both certified robust errors and empirical robust errors.
  arXiv  Detail & Related papers  (2020-06-12T09:17:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.