Scenario-based Evaluation of Prediction Models for Automated Vehicles

• URL: http://arxiv.org/abs/2210.06553v1
• Date: Tue, 11 Oct 2022 09:45:43 GMT
• Title: Scenario-based Evaluation of Prediction Models for Automated Vehicles
• Authors: Manuel Mu\~noz S\'anchez, Jos Elfring, Emilia Silvas and Ren\'e van de Molengraft
• Abstract summary: We argue that following evaluation practices in safety assessment for automated vehicles should be performed in a scenario-based fashion. We categorize trajectories of Open Motion dataset according to the type of movement they capture. Results show that common evaluation methods are insufficient and the assessment should be performed depending on the application in which the model will operate.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: To operate safely, an automated vehicle (AV) must anticipate how the environment around it will evolve. For that purpose, it is important to know which prediction models are most appropriate for every situation. Currently, assessment of prediction models is often performed over a set of trajectories without distinction of the type of movement they capture, resulting in the inability to determine the suitability of each model for different situations. In this work we illustrate how standardized evaluation methods result in wrong conclusions regarding a model's predictive capabilities, preventing a clear assessment of prediction models and potentially leading to dangerous on-road situations. We argue that following evaluation practices in safety assessment for AVs, assessment of prediction models should be performed in a scenario-based fashion. To encourage scenario-based assessment of prediction models and illustrate the dangers of improper assessment, we categorize trajectories of the Waymo Open Motion dataset according to the type of movement they capture. Next, three different models are thoroughly evaluated for different trajectory types and prediction horizons. Results show that common evaluation methods are insufficient and the assessment should be performed depending on the application in which the model will operate.

Related papers

• Predictive Churn with the Set of Good Models [64.05949860750235]
  We study the effect of conflicting predictions over the set of near-optimal machine learning models. We present theoretical results on the expected churn between models within the Rashomon set. We show how our approach can be used to better anticipate, reduce, and avoid churn in consumer-facing applications.
  arXiv  Detail & Related papers  (2024-02-12T16:15:25Z)
• Evaluating Pedestrian Trajectory Prediction Methods with Respect to Autonomous Driving [0.9217021281095907]
  In this paper, we assess the state of the art in pedestrian trajectory prediction within the context of generating single trajectories. The evaluation is conducted on the widely-used ETH/UCY dataset where the Average Displacement Error (ADE) and the Final Displacement Error (FDE) are reported.
  arXiv  Detail & Related papers  (2023-08-09T19:21:50Z)
• Streaming egocentric action anticipation: An evaluation scheme and approach [27.391434284586985]
  Egocentric action anticipation aims to predict the future actions the camera wearer will perform from the observation of the past. Current evaluation schemes assume that predictions are available right after the input video is observed. We propose a streaming egocentric action evaluation scheme which assumes that predictions are performed online and made available only after the model has processed the current input segment.
  arXiv  Detail & Related papers  (2023-06-29T04:53:29Z)
• Model Predictive Control with Gaussian-Process-Supported Dynamical Constraints for Autonomous Vehicles [82.65261980827594]
  We propose a model predictive control approach for autonomous vehicles that exploits learned Gaussian processes for predicting human driving behavior. A multi-mode predictive control approach considers the possible intentions of the human drivers.
  arXiv  Detail & Related papers  (2023-03-08T17:14:57Z)
• Benchmark for Models Predicting Human Behavior in Gap Acceptance Scenarios [4.801975818473341]
  We develop a framework facilitating the evaluation of any model, by any metric, and in any scenario. We then apply this framework to state-of-the-art prediction models, which all show themselves to be unreliable in the most safety-critical situations.
  arXiv  Detail & Related papers  (2022-11-10T09:59:38Z)
• Uncertainty estimation of pedestrian future trajectory using Bayesian approximation [137.00426219455116]
  Under dynamic traffic scenarios, planning based on deterministic predictions is not trustworthy. The authors propose to quantify uncertainty during forecasting using approximation which deterministic approaches fail to capture. The effect of dropout weights and long-term prediction on future state uncertainty has been studied.
  arXiv  Detail & Related papers  (2022-05-04T04:23:38Z)
• Forecast Evaluation in Large Cross-Sections of Realized Volatility [0.0]
  We evaluate the predictive accuracy of the model based on the augmented cross-section when forecasting Realized volatility. We study the sensitivity of forecasts to the model specification by incorporating a measurement error correction as well as cross-sectional jump component measures.
  arXiv  Detail & Related papers  (2021-12-09T13:19:09Z)
• Acting upon Imagination: when to trust imagined trajectories in model based reinforcement learning [1.26990070983988]
  Model-based reinforcement learning (MBRL) aims to learn model(s) of the environment dynamics that can predict the outcome of its actions. We propose uncertainty estimation methods for online evaluation of imagined trajectories. Results highlight significant reduction on computational costs without sacrificing performance.
  arXiv  Detail & Related papers  (2021-05-12T15:04:07Z)
• Characterizing Fairness Over the Set of Good Models Under Selective Labels [69.64662540443162]
  We develop a framework for characterizing predictive fairness properties over the set of models that deliver similar overall performance. We provide tractable algorithms to compute the range of attainable group-level predictive disparities. We extend our framework to address the empirically relevant challenge of selectively labelled data.
  arXiv  Detail & Related papers  (2021-01-02T02:11:37Z)
• Forethought and Hindsight in Credit Assignment [62.05690959741223]
  We work to understand the gains and peculiarities of planning employed as forethought via forward models or as hindsight operating with backward models. We investigate the best use of models in planning, primarily focusing on the selection of states in which predictions should be (re)-evaluated.
  arXiv  Detail & Related papers  (2020-10-26T16:00:47Z)
• Counterfactual Predictions under Runtime Confounding [74.90756694584839]
  We study the counterfactual prediction task in the setting where all relevant factors are captured in the historical data. We propose a doubly-robust procedure for learning counterfactual prediction models in this setting.
  arXiv  Detail & Related papers  (2020-06-30T15:49:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.