Realistic Adversarial Attacks for Robustness Evaluation of Trajectory Prediction Models via Future State Perturbation

• URL: http://arxiv.org/abs/2505.06134v1
• Date: Fri, 09 May 2025 15:40:32 GMT
• Title: Realistic Adversarial Attacks for Robustness Evaluation of Trajectory Prediction Models via Future State Perturbation
• Authors: Julian F. Schumann, Jeroen Hagenus, Frederik Baymler Mathiesen, Arkady Zgonnikov,
• Abstract summary: Trajectory prediction is a key element of autonomous vehicle systems.<n>Current approaches tend to focus on perturbing the past positions of surrounding agents.<n>We introduce new performance measures to assess the realism and impact of these adversarial trajectories.
• Score: 1.124958340749622
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Trajectory prediction is a key element of autonomous vehicle systems, enabling them to anticipate and react to the movements of other road users. Evaluating the robustness of prediction models against adversarial attacks is essential to ensure their reliability in real-world traffic. However, current approaches tend to focus on perturbing the past positions of surrounding agents, which can generate unrealistic scenarios and overlook critical vulnerabilities. This limitation may result in overly optimistic assessments of model performance in real-world conditions. In this work, we demonstrate that perturbing not just past but also future states of adversarial agents can uncover previously undetected weaknesses and thereby provide a more rigorous evaluation of model robustness. Our novel approach incorporates dynamic constraints and preserves tactical behaviors, enabling more effective and realistic adversarial attacks. We introduce new performance measures to assess the realism and impact of these adversarial trajectories. Testing our method on a state-of-the-art prediction model revealed significant increases in prediction errors and collision rates under adversarial conditions. Qualitative analysis further showed that our attacks can expose critical weaknesses, such as the inability of the model to detect potential collisions in what appear to be safe predictions. These results underscore the need for more comprehensive adversarial testing to better evaluate and improve the reliability of trajectory prediction models for autonomous vehicles.

Related papers

• Preliminary Investigation into Uncertainty-Aware Attack Stage Classification [81.28215542218724]
  This work addresses the problem of attack stage inference under uncertainty.<n>We propose a classification approach based on Evidential Deep Learning (EDL), which models predictive uncertainty by outputting parameters of a Dirichlet distribution over possible stages.<n>Preliminary experiments in a simulated environment demonstrate that the proposed model can accurately infer the stage of an attack with confidence.
  arXiv  Detail & Related papers  (2025-08-01T06:58:00Z)
• Certified Human Trajectory Prediction [66.1736456453465]
  Tray prediction plays an essential role in autonomous vehicles. We propose a certification approach tailored for the task of trajectory prediction. We address the inherent challenges associated with trajectory prediction, including unbounded outputs, and mutli-modality.
  arXiv  Detail & Related papers  (2024-03-20T17:41:35Z)
• SAFE-SIM: Safety-Critical Closed-Loop Traffic Simulation with Diffusion-Controllable Adversaries [94.84458417662407]
  We introduce SAFE-SIM, a controllable closed-loop safety-critical simulation framework. Our approach yields two distinct advantages: 1) generating realistic long-tail safety-critical scenarios that closely reflect real-world conditions, and 2) providing controllable adversarial behavior for more comprehensive and interactive evaluations. We validate our framework empirically using the nuScenes and nuPlan datasets across multiple planners, demonstrating improvements in both realism and controllability.
  arXiv  Detail & Related papers  (2023-12-31T04:14:43Z)
• Learn from the Past: A Proxy Guided Adversarial Defense Framework with Self Distillation Regularization [53.04697800214848]
  Adversarial Training (AT) is pivotal in fortifying the robustness of deep learning models. AT methods, relying on direct iterative updates for target model's defense, frequently encounter obstacles such as unstable training and catastrophic overfitting. We present a general proxy guided defense framework, LAST' (bf Learn from the Pbf ast)
  arXiv  Detail & Related papers  (2023-10-19T13:13:41Z)
• Adversarial Backdoor Attack by Naturalistic Data Poisoning on Trajectory Prediction in Autonomous Driving [18.72382517467458]
  We propose a novel adversarial backdoor attack against trajectory prediction models. Our attack affects the victim at training time via naturalistic, hence stealthy, poisoned samples crafted using a novel two-step approach. We show that the proposed attack is highly effective, as it can significantly hinder the performance of prediction models.
  arXiv  Detail & Related papers  (2023-06-27T19:15:06Z)
• Consistent Valid Physically-Realizable Adversarial Attack against Crowd-flow Prediction Models [4.286570387250455]
  deep learning (DL) models can effectively learn city-wide crowd-flow patterns. DL models have been known to perform poorly on inconspicuous adversarial perturbations.
  arXiv  Detail & Related papers  (2023-03-05T13:30:25Z)
• Benchmark for Models Predicting Human Behavior in Gap Acceptance Scenarios [4.801975818473341]
  We develop a framework facilitating the evaluation of any model, by any metric, and in any scenario. We then apply this framework to state-of-the-art prediction models, which all show themselves to be unreliable in the most safety-critical situations.
  arXiv  Detail & Related papers  (2022-11-10T09:59:38Z)
• AdvDO: Realistic Adversarial Attacks for Trajectory Prediction [87.96767885419423]
  Trajectory prediction is essential for autonomous vehicles to plan correct and safe driving behaviors. We devise an optimization-based adversarial attack framework to generate realistic adversarial trajectories. Our attack can lead an AV to drive off road or collide into other vehicles in simulation.
  arXiv  Detail & Related papers  (2022-09-19T03:34:59Z)
• Semi-supervised Semantics-guided Adversarial Training for Trajectory Prediction [15.707419899141698]
  Adversarial attacks on trajectory prediction may mislead the prediction of future trajectories and induce unsafe planning. We present a novel adversarial training method for trajectory prediction. Our method can effectively mitigate the impact of adversarial attacks by up to 73% and outperform other popular defense methods.
  arXiv  Detail & Related papers  (2022-05-27T20:50:36Z)
• Heterogeneous-Agent Trajectory Forecasting Incorporating Class Uncertainty [54.88405167739227]
  We present HAICU, a method for heterogeneous-agent trajectory forecasting that explicitly incorporates agents' class probabilities. We additionally present PUP, a new challenging real-world autonomous driving dataset. We demonstrate that incorporating class probabilities in trajectory forecasting significantly improves performance in the face of uncertainty.
  arXiv  Detail & Related papers  (2021-04-26T10:28:34Z)
• Trust but Verify: Assigning Prediction Credibility by Counterfactual Constrained Learning [123.3472310767721]
  Prediction credibility measures are fundamental in statistics and machine learning. These measures should account for the wide variety of models used in practice. The framework developed in this work expresses the credibility as a risk-fit trade-off.
  arXiv  Detail & Related papers  (2020-11-24T19:52:38Z)
• The Importance of Prior Knowledge in Precise Multimodal Prediction [71.74884391209955]
  Roads have well defined geometries, topologies, and traffic rules. In this paper we propose to incorporate structured priors as a loss function. We demonstrate the effectiveness of our approach on real-world self-driving datasets.
  arXiv  Detail & Related papers  (2020-06-04T03:56:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.