Beyond RMSE: Do machine-learned models of road user interaction produce human-like behavior?

• URL: http://arxiv.org/abs/2206.11110v2
• Date: Tue, 28 Mar 2023 18:38:59 GMT
• Title: Beyond RMSE: Do machine-learned models of road user interaction produce human-like behavior?
• Authors: Aravinda Ramakrishnan Srinivasan, Yi-Shin Lin, Morris Antonello, Anthony Knittel, Mohamed Hasan, Majd Hawasly, John Redford, Subramanian Ramamoorthy, Matteo Leonetti, Jac Billington, Richard Romano, Gustav Markkula
• Abstract summary: We introduce quantitative metrics to demonstrate presence of three different behavioral phenomena in a naturalistic highway driving dataset. We analyze the behavior of three machine-learned models using the same metrics.
• Score: 12.378231329297137
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Autonomous vehicles use a variety of sensors and machine-learned models to predict the behavior of surrounding road users. Most of the machine-learned models in the literature focus on quantitative error metrics like the root mean square error (RMSE) to learn and report their models' capabilities. This focus on quantitative error metrics tends to ignore the more important behavioral aspect of the models, raising the question of whether these models really predict human-like behavior. Thus, we propose to analyze the output of machine-learned models much like we would analyze human data in conventional behavioral research. We introduce quantitative metrics to demonstrate presence of three different behavioral phenomena in a naturalistic highway driving dataset: 1) The kinematics-dependence of who passes a merging point first 2) Lane change by an on-highway vehicle to accommodate an on-ramp vehicle 3) Lane changes by vehicles on the highway to avoid lead vehicle conflicts. Then, we analyze the behavior of three machine-learned models using the same metrics. Even though the models' RMSE value differed, all the models captured the kinematic-dependent merging behavior but struggled at varying degrees to capture the more nuanced courtesy lane change and highway lane change behavior. Additionally, the collision aversion analysis during lane changes showed that the models struggled to capture the physical aspect of human driving: leaving adequate gap between the vehicles. Thus, our analysis highlighted the inadequacy of simple quantitative metrics and the need to take a broader behavioral perspective when analyzing machine-learned models of human driving predictions.

Related papers

• MetaFollower: Adaptable Personalized Autonomous Car Following [63.90050686330677]
  We propose an adaptable personalized car-following framework - MetaFollower. We first utilize Model-Agnostic Meta-Learning (MAML) to extract common driving knowledge from various CF events. We additionally combine Long Short-Term Memory (LSTM) and Intelligent Driver Model (IDM) to reflect temporal heterogeneity with high interpretability.
  arXiv  Detail & Related papers  (2024-06-23T15:30:40Z)
• Trajeglish: Traffic Modeling as Next-Token Prediction [67.28197954427638]
  A longstanding challenge for self-driving development is simulating dynamic driving scenarios seeded from recorded driving logs. We apply tools from discrete sequence modeling to model how vehicles, pedestrians and cyclists interact in driving scenarios. Our model tops the Sim Agents Benchmark, surpassing prior work along the realism meta metric by 3.3% and along the interaction metric by 9.9%.
  arXiv  Detail & Related papers  (2023-12-07T18:53:27Z)
• On Learning the Tail Quantiles of Driving Behavior Distributions via Quantile Regression and Flows [13.540998552232006]
  We consider the problem of learning models that accurately capture the diversity and tail quantiles of human driver behavior probability distributions. We adapt two flexible quantile learning frameworks for this setting that avoid strong distributional assumptions. We evaluate our approach in a one-step acceleration prediction task, and in multi-step driver simulation rollouts.
  arXiv  Detail & Related papers  (2023-05-22T15:09:04Z)
• Cross or Wait? Predicting Pedestrian Interaction Outcomes at Unsignalized Crossings [12.800489342833275]
  We use machine learning to predict pedestrian crossing behavior including pedestrian crossing decision, crossing initiation time (CIT), and crossing duration (CD) when interacting with vehicles at unsignalized crossings. Compared with the logistic regression baseline model, our proposed neural network model improves the prediction accuracy and F1 score by 4.46% and 3.23%, respectively. Our model also reduces the root mean squared error (RMSE) for CIT and CD by 21.56% and 30.14% compared with the linear regression model.
  arXiv  Detail & Related papers  (2023-04-17T13:20:04Z)
• Robustness Benchmark of Road User Trajectory Prediction Models for Automated Driving [0.0]
  We benchmark machine learning models against perturbations that simulate functional insufficiencies observed during model deployment in a vehicle. Training the models with similar perturbations effectively reduces performance degradation, with error increases of up to +87.5%. We argue that despite being an effective mitigation strategy, data augmentation through perturbations during training does not guarantee robustness towards unforeseen perturbations.
  arXiv  Detail & Related papers  (2023-04-04T15:47:42Z)
• Modeling human road crossing decisions as reward maximization with visual perception limitations [23.561752465516047]
  We develop a model of human pedestrian crossing decisions based on computational rationality. We show that the proposed cognitive-RL model captures human-like patterns of gap acceptance and crossing initiation time. Our results suggest that this is instead a rational adaption to human perceptual limitations.
  arXiv  Detail & Related papers  (2023-01-27T14:20:35Z)
• Comparing merging behaviors observed in naturalistic data with behaviors generated by a machine learned model [4.879725885276143]
  We study highway driving as an example scenario, and introduce metrics to quantitatively demonstrate the presence of two familiar behavioral phenomena. Applying the exact same metrics to the output of a state-of-the-art machine-learned model, we show that the model is capable of reproducing the former phenomenon, but not the latter.
  arXiv  Detail & Related papers  (2021-04-21T12:31:29Z)
• A model for traffic incident prediction using emergency braking data [77.34726150561087]
  We address the fundamental problem of data scarcity in road traffic accident prediction by training our model on emergency braking events instead of accidents. We present a prototype implementing a traffic incident prediction model for Germany based on emergency braking data from Mercedes-Benz vehicles.
  arXiv  Detail & Related papers  (2021-02-12T18:17:12Z)
• Fine-Grained Vehicle Perception via 3D Part-Guided Visual Data Augmentation [77.60050239225086]
  We propose an effective training data generation process by fitting a 3D car model with dynamic parts to vehicles in real images. Our approach is fully automatic without any human interaction. We present a multi-task network for VUS parsing and a multi-stream network for VHI parsing.
  arXiv  Detail & Related papers  (2020-12-15T03:03:38Z)
• Detecting 32 Pedestrian Attributes for Autonomous Vehicles [103.87351701138554]
  In this paper, we address the problem of jointly detecting pedestrians and recognizing 32 pedestrian attributes. We introduce a Multi-Task Learning (MTL) model relying on a composite field framework, which achieves both goals in an efficient way. We show competitive detection and attribute recognition results, as well as a more stable MTL training.
  arXiv  Detail & Related papers  (2020-12-04T15:10:12Z)
• Learning Accurate and Human-Like Driving using Semantic Maps and Attention [152.48143666881418]
  This paper investigates how end-to-end driving models can be improved to drive more accurately and human-like. We exploit semantic and visual maps from HERE Technologies and augment the existing Drive360 dataset with such. Our models are trained and evaluated on the Drive360 + HERE dataset, which features 60 hours and 3000 km of real-world driving data.
  arXiv  Detail & Related papers  (2020-07-10T22:25:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.