Attacking Autonomous Driving Agents with Adversarial Machine Learning: A Holistic Evaluation with the CARLA Leaderboard

• URL: http://arxiv.org/abs/2511.14876v1
• Date: Tue, 18 Nov 2025 19:49:46 GMT
• Title: Attacking Autonomous Driving Agents with Adversarial Machine Learning: A Holistic Evaluation with the CARLA Leaderboard
• Authors: Henry Wong, Clement Fung, Weiran Lin, Karen Li, Stanley Chen, Lujo Bauer,
• Abstract summary: We evaluate attacks against a variety of driving agents, rather than against ML models in isolation.<n>We create adversarial patches designed to stop or steer driving agents, stream them into the CARLA simulator at runtime, and evaluate them against agents from the CARLA Leaderboard.<n>We show that, although some attacks can successfully mislead ML models into predicting erroneous stopping or steering commands, some driving agents use modules, such as PID control or GPS-based rules, that can overrule attacker-manipulated predictions from ML models.
• Score: 8.45711173703347
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: To autonomously control vehicles, driving agents use outputs from a combination of machine-learning (ML) models, controller logic, and custom modules. Although numerous prior works have shown that adversarial examples can mislead ML models used in autonomous driving contexts, it remains unclear if these attacks are effective at producing harmful driving actions for various agents, environments, and scenarios. To assess the risk of adversarial examples to autonomous driving, we evaluate attacks against a variety of driving agents, rather than against ML models in isolation. To support this evaluation, we leverage CARLA, an urban driving simulator, to create and evaluate adversarial examples. We create adversarial patches designed to stop or steer driving agents, stream them into the CARLA simulator at runtime, and evaluate them against agents from the CARLA Leaderboard, a public repository of best-performing autonomous driving agents from an annual research competition. Unlike prior work, we evaluate attacks against autonomous driving systems without creating or modifying any driving-agent code and against all parts of the agent included with the ML model. We perform a case-study investigation of two attack strategies against three open-source driving agents from the CARLA Leaderboard across multiple driving scenarios, lighting conditions, and locations. Interestingly, we show that, although some attacks can successfully mislead ML models into predicting erroneous stopping or steering commands, some driving agents use modules, such as PID control or GPS-based rules, that can overrule attacker-manipulated predictions from ML models.

Related papers

• Dynamic Deception: When Pedestrians Team Up to Fool Autonomous Cars [7.072654753416604]
  adversarial attacks on autonomous-driving perception models fail to cause system-level failures once deployed in a full driving stack.<n>We introduce a system-level attack in which multiple dynamic elements carry adversarial patches.<n>We evaluate our attacks in the CARLA simulator using a state-of-the-art autonomous driving agent.
  arXiv  Detail & Related papers  (2026-02-20T09:09:24Z)
• Agent2Agent Threats in Safety-Critical LLM Assistants: A Human-Centric Taxonomy [4.058281338403478]
  We propose a threat modeling framework called AgentHeLLM that separates asset identification from attack path analysis.<n>We introduce a human-centric asset taxonomy derived from harm-oriented "victim modeling" and inspired by the Universal Declaration of Human Rights.<n>We demonstrate the framework's practical applicability through an open-source attack path suggestion tool AgentHeLLM Attack Path Generator.
  arXiv  Detail & Related papers  (2026-02-05T16:53:41Z)
• SPACeR: Self-Play Anchoring with Centralized Reference Models [50.55045557371374]
  Sim agent policies are realistic, human-like, fast, and scalable in multi-agent settings.<n>Recent progress in imitation learning with large diffusion-based or tokenized models has shown that behaviors can be captured directly from human driving data.<n>We propose SPACeR, a framework that leverages a pretrained tokenized autoregressive motion model as a central reference policy.
  arXiv  Detail & Related papers  (2025-10-20T19:53:02Z)
• Adaptive Attacks on Trusted Monitors Subvert AI Control Protocols [80.68060125494645]
  We study adaptive attacks by an untrusted model that knows the protocol and the monitor model.<n>We instantiate a simple adaptive attack vector by which the attacker embeds publicly known or zero-shot prompt injections in the model outputs.
  arXiv  Detail & Related papers  (2025-10-10T15:12:44Z)
• CHARMS: A Cognitive Hierarchical Agent for Reasoning and Motion Stylization in Autonomous Driving [7.672737334176452]
  This paper proposes a Cognitive Hierarchical Agent for Reasoning and Motion Stylization (CHARMS)<n>CHARMS captures human-like reasoning patterns through a two-stage training pipeline comprising reinforcement learning pretraining and supervised fine-tuning.<n>It is capable of both making intelligent driving decisions as an ego vehicle and generating diverse, realistic driving scenarios as environment vehicles.
  arXiv  Detail & Related papers  (2025-04-03T10:15:19Z)
• ManeuverGPT Agentic Control for Safe Autonomous Stunt Maneuvers [0.0]
  This paper presents a novel framework, ManeuverGPT, for generating and executing high-dynamic stunt maneuvers in autonomous vehicles.<n>We propose an agentic architecture comprised of three specialized agents.<n> Experimental results demonstrate successful J-turn execution across multiple vehicle models.
  arXiv  Detail & Related papers  (2025-03-12T03:51:41Z)
• SafeAuto: Knowledge-Enhanced Safe Autonomous Driving with Multimodal Foundation Models [63.71984266104757]
  We propose SafeAuto, a framework that enhances MLLM-based autonomous driving by incorporating both unstructured and structured knowledge.<n>To explicitly integrate safety knowledge, we develop a reasoning component that translates traffic rules into first-order logic.<n>Our Multimodal Retrieval-Augmented Generation model leverages video, control signals, and environmental attributes to learn from past driving experiences.
  arXiv  Detail & Related papers  (2025-02-28T21:53:47Z)
• Work-in-Progress: Crash Course: Can (Under Attack) Autonomous Driving Beat Human Drivers? [60.51287814584477]
  This paper evaluates the inherent risks in autonomous driving by examining the current landscape of AVs. We develop specific claims highlighting the delicate balance between the advantages of AVs and potential security challenges in real-world scenarios.
  arXiv  Detail & Related papers  (2024-05-14T09:42:21Z)
• Physical Backdoor Attack can Jeopardize Driving with Vision-Large-Language Models [53.701148276912406]
  Vision-Large-Language-models (VLMs) have great application prospects in autonomous driving. BadVLMDriver is the first backdoor attack against VLMs for autonomous driving that can be launched in practice using physical objects. BadVLMDriver achieves a 92% attack success rate in inducing a sudden acceleration when coming across a pedestrian holding a red balloon.
  arXiv  Detail & Related papers  (2024-04-19T14:40:38Z)
• DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving [69.82743399946371]
  DriveMLM is a framework that can perform close-loop autonomous driving in realistic simulators. We employ a multi-modal LLM (MLLM) to model the behavior planning module of a module AD system. This model can plug-and-play in existing AD systems such as Apollo for close-loop driving.
  arXiv  Detail & Related papers  (2023-12-14T18:59:05Z)
• Attacks and Faults Injection in Self-Driving Agents on the Carla Simulator -- Experience Report [1.933681537640272]
  We report on the injection of adversarial attacks and software faults in a self-driving agent running in a driving simulator. We show that adversarial attacks and faults injected in the trained agent can lead to erroneous decisions and severely jeopardize safety.
  arXiv  Detail & Related papers  (2022-02-25T21:46:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.