Injecting Planning-Awareness into Prediction and Detection Evaluation
- URL: http://arxiv.org/abs/2110.03270v1
- Date: Thu, 7 Oct 2021 08:52:48 GMT
- Title: Injecting Planning-Awareness into Prediction and Detection Evaluation
- Authors: Boris Ivanovic, Marco Pavone
- Abstract summary: We take a step back and critically assess current evaluation metrics, proposing task-aware metrics as a better measure of performance in systems where they are deployed.
Experiments on an illustrative simulation as well as real-world autonomous driving data validate that our proposed task-aware metrics are able to account for outcome asymmetry and provide a better estimate of a model's closed-loop performance.
- Score: 42.228191984697006
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Detecting other agents and forecasting their behavior is an integral part of
the modern robotic autonomy stack, especially in safety-critical scenarios
entailing human-robot interaction such as autonomous driving. Due to the
importance of these components, there has been a significant amount of interest
and research in perception and trajectory forecasting, resulting in a wide
variety of approaches. Common to most works, however, is the use of the same
few accuracy-based evaluation metrics, e.g., intersection-over-union,
displacement error, log-likelihood, etc. While these metrics are informative,
they are task-agnostic and outputs that are evaluated as equal can lead to
vastly different outcomes in downstream planning and decision making. In this
work, we take a step back and critically assess current evaluation metrics,
proposing task-aware metrics as a better measure of performance in systems
where they are deployed. Experiments on an illustrative simulation as well as
real-world autonomous driving data validate that our proposed task-aware
metrics are able to account for outcome asymmetry and provide a better estimate
of a model's closed-loop performance.
Related papers
- JRDB-Traj: A Dataset and Benchmark for Trajectory Forecasting in Crowds [79.00975648564483]
Trajectory forecasting models, employed in fields such as robotics, autonomous vehicles, and navigation, face challenges in real-world scenarios.
This dataset provides comprehensive data, including the locations of all agents, scene images, and point clouds, all from the robot's perspective.
The objective is to predict the future positions of agents relative to the robot using raw sensory input data.
arXiv Detail & Related papers (2023-11-05T18:59:31Z) - From Static Benchmarks to Adaptive Testing: Psychometrics in AI Evaluation [60.14902811624433]
We discuss a paradigm shift from static evaluation methods to adaptive testing.
This involves estimating the characteristics and value of each test item in the benchmark and dynamically adjusting items in real-time.
We analyze the current approaches, advantages, and underlying reasons for adopting psychometrics in AI evaluation.
arXiv Detail & Related papers (2023-06-18T09:54:33Z) - A Control-Centric Benchmark for Video Prediction [69.22614362800692]
We propose a benchmark for action-conditioned video prediction in the form of a control benchmark.
Our benchmark includes simulated environments with 11 task categories and 310 task instance definitions.
We then leverage our benchmark to study the effects of scaling model size, quantity of training data, and model ensembling.
arXiv Detail & Related papers (2023-04-26T17:59:45Z) - Towards trustworthy multi-modal motion prediction: Holistic evaluation
and interpretability of outputs [3.5240925434839054]
We focus on evaluation criteria, robustness, and interpretability of outputs.
We propose an intent prediction layer that can be attached to multi-modal motion prediction models.
arXiv Detail & Related papers (2022-10-28T14:14:22Z) - Architectural patterns for handling runtime uncertainty of data-driven
models in safety-critical perception [1.7616042687330642]
We present additional architectural patterns for handling uncertainty estimation.
We evaluate the four patterns qualitatively and quantitatively with respect to safety and performance gains.
We conclude that the consideration of context information of the driving situation makes it possible to accept more or less uncertainty depending on the inherent risk of the situation.
arXiv Detail & Related papers (2022-06-14T13:31:36Z) - Control-Aware Prediction Objectives for Autonomous Driving [78.19515972466063]
We present control-aware prediction objectives (CAPOs) to evaluate the downstream effect of predictions on control without requiring the planner be differentiable.
We propose two types of importance weights that weight the predictive likelihood: one using an attention model between agents, and another based on control variation when exchanging predicted trajectories for ground truth trajectories.
arXiv Detail & Related papers (2022-04-28T07:37:21Z) - Rethinking Trajectory Forecasting Evaluation [42.228191984697006]
We take a step back and critically evaluate current trajectory forecasting metrics.
We propose task-aware metrics as a better measure of performance in systems where prediction is being deployed.
arXiv Detail & Related papers (2021-07-21T18:20:03Z) - Uncertainty-Aware Vehicle Orientation Estimation for Joint
Detection-Prediction Models [12.56249869551208]
Orientation is an important property for downstream modules of an autonomous system.
We present a method that extends the existing models that perform joint object detection and motion prediction.
In addition, the approach is able to quantify prediction uncertainty, outputting the probability that the inferred orientation is flipped.
arXiv Detail & Related papers (2020-11-05T21:59:44Z) - Modeling Perception Errors towards Robust Decision Making in Autonomous
Vehicles [11.503090828741191]
We propose a simulation-based methodology towards answering the question: is a perception subsystem sufficient for the decision making subsystem to make robust, safe decisions?
We show how to analyze the impact of different kinds of sensing and perception errors on the behavior of the autonomous system.
arXiv Detail & Related papers (2020-01-31T08:02:14Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.