The Autonomy Tax: Defense Training Breaks LLM Agents

• URL: http://arxiv.org/abs/2603.19423v1
• Date: Thu, 19 Mar 2026 19:33:17 GMT
• Title: The Autonomy Tax: Defense Training Breaks LLM Agents
• Abstract summary: Defense training designed to improve safety systematically destroys agent competence while failing to prevent sophisticated attacks.<n>We evaluate defended models against undefended baselines across 97 agent tasks and 1,000 adversarial prompts.<n>Our findings demonstrate that current defense paradigms optimize for single-turn refusal benchmarks while rendering multi-step agents fundamentally unreliable.
• Score: 5.990318568221089
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language model (LLM) agents increasingly rely on external tools (file operations, API calls, database transactions) to autonomously complete complex multi-step tasks. Practitioners deploy defense-trained models to protect against prompt injection attacks that manipulate agent behavior through malicious observations or retrieved content. We reveal a fundamental \textbf{capability-alignment paradox}: defense training designed to improve safety systematically destroys agent competence while failing to prevent sophisticated attacks. Evaluating defended models against undefended baselines across 97 agent tasks and 1,000 adversarial prompts, we uncover three systematic biases unique to multi-step agents. \textbf{Agent incompetence bias} manifests as immediate tool execution breakdown, with models refusing or generating invalid actions on benign tasks before observing any external content. \textbf{Cascade amplification bias} causes early failures to propagate through retry loops, pushing defended models to timeout on 99\% of tasks compared to 13\% for baselines. \textbf{Trigger bias} leads to paradoxical security degradation where defended models perform worse than undefended baselines while straightforward attacks bypass defenses at high rates. Root cause analysis reveals these biases stem from shortcut learning: models overfit to surface attack patterns rather than semantic threat understanding, evidenced by extreme variance in defense effectiveness across attack categories. Our findings demonstrate that current defense paradigms optimize for single-turn refusal benchmarks while rendering multi-step agents fundamentally unreliable, necessitating new approaches that preserve tool execution competence under adversarial conditions.

Related papers

• Defending against Adaptive Prompt Injection Attacks via Reasoning-enabled Task Alignment [25.752599132396437]
  Indirect prompt injection attacks hijack LLM-based agents by embedding malicious instructions in third-party data that the agent retrieves during task execution.<n>Existing defenses report near-zero attack success rate on static benchmarks, yet recent adaptive evaluations show that these results collapse once the attacker is allowed to optimize against the deployed defense.<n>We propose RETA, a training-based method that grounds defense decisions on the user tasks rather than attacker-controlled data.
  arXiv  Detail & Related papers  (2026-06-13T19:15:44Z)
• AgentVisor: Defending LLM Agents Against Prompt Injection via Semantic Virtualization [37.5130864321051]
  We propose AgentVisor, a novel defense framework that enforces semantic privilege separation.<n>AgentVisor treats the target agent as an untrusted guest and intercepts tool calls via a trusted semantic visor.<n>We show that AgentVisor reduces the attack success rate to 0.65%, achieving this strong defense while incurring only a 1.45% average decrease in utility.
  arXiv  Detail & Related papers  (2026-04-27T07:12:52Z)
• PlanGuard: Defending Agents against Indirect Prompt Injection via Planning-based Consistency Verification [1.7904458681854372]
  PlanGuard is a training-free defense framework based on the principle of Context Isolation.<n>PlanGuard effectively neutralizes attacks, reducing the Attack Success Rate (ASR) from 72.8% to 0%.
  arXiv  Detail & Related papers  (2026-04-11T09:59:46Z)
• Learning When to Act or Refuse: Guarding Agentic Reasoning Models for Safe Multi-Step Tool Use [6.622648583261088]
  Agentic language models must plan, call tools, and execute long-horizon actions where a single misstep can cause irreversible harm.<n>We introduce MOSAIC, a framework that aligns agents for safe multi-step tool use by making safety decisions explicit and learnable.<n>We show that MOSAIC reduces harmful behavior by up to 50%, increases harmful-task refusal by over 20% on injection attacks, cuts privacy leakage, and preserves or improves benign task performance.
  arXiv  Detail & Related papers  (2026-03-03T17:59:35Z)
• TraceGuard: Process-Guided Firewall against Reasoning Backdoors in Large Language Models [19.148124494194317]
  We propose TraceGuard, a process-guided security framework that transforms small-scale models into robust reasoning firewalls.<n>Our approach treats the reasoning trace as an untrusted payload and establishes a defense-in-depth strategy.<n>We demonstrate robustness against adaptive adversaries in a grey-box setting, establishing TraceGuard as a viable, low-latency security primitive.
  arXiv  Detail & Related papers  (2026-03-02T22:19:13Z)
• AgentSentry: Mitigating Indirect Prompt Injection in LLM Agents via Temporal Causal Diagnostics and Context Purification [25.817251923574286]
  We propose a novel inference-time detection and mitigation framework for large language model (LLM) agents.<n>AgentSentry is the first inference-time defense to model multi-turn IPI as a temporal causal takeover.<n>We evaluate AgentSentry on the textscAgentDojo benchmark across four task suites, three IPI attack families, and multiple black-box LLMs.
  arXiv  Detail & Related papers  (2026-02-26T07:59:10Z)
• Defenses Against Prompt Attacks Learn Surface Heuristics [40.392588465939106]
  Large language models (LLMs) are increasingly deployed in security-sensitive applications.<n>LLMs may override intended logic when adversarial instructions appear in user queries or retrieved content.<n>Recent defenses rely on supervised fine-tuning with benign and malicious labels.
  arXiv  Detail & Related papers  (2026-01-12T04:12:48Z)
• BackdoorAgent: A Unified Framework for Backdoor Attacks on LLM-based Agents [58.83028403414688]
  Large language model (LLM) agents execute tasks through multi-step workflow that combine planning, memory, and tool use.<n>Backdoor triggers injected into specific stages of an agent workflow can persist through multiple intermediate states and adversely influence downstream outputs.<n>We propose textbfBackdoorAgent, a modular and stage-aware framework that provides a unified agent-centric view of backdoor threats in LLM agents.
  arXiv  Detail & Related papers  (2026-01-08T03:49:39Z)
• Malice in Agentland: Down the Rabbit Hole of Backdoors in the AI Supply Chain [82.98626829232899]
  Fine-tuning AI agents on data from their own interactions introduces a critical security vulnerability within the AI supply chain.<n>We show that adversaries can easily poison the data collection pipeline to embed hard-to-detect backdoors.
  arXiv  Detail & Related papers  (2025-10-03T12:47:21Z)
• BlindGuard: Safeguarding LLM-based Multi-Agent Systems under Unknown Attacks [58.959622170433725]
  BlindGuard is an unsupervised defense method that learns without requiring any attack-specific labels or prior knowledge of malicious behaviors.<n>We show that BlindGuard effectively detects diverse attack types (i.e., prompt injection, memory poisoning, and tool attack) across multi-agent systems.
  arXiv  Detail & Related papers  (2025-08-11T16:04:47Z)
• MISLEADER: Defending against Model Extraction with Ensembles of Distilled Models [56.09354775405601]
  Model extraction attacks aim to replicate the functionality of a black-box model through query access.<n>Most existing defenses presume that attacker queries have out-of-distribution (OOD) samples, enabling them to detect and disrupt suspicious inputs.<n>We propose MISLEADER, a novel defense strategy that does not rely on OOD assumptions.
  arXiv  Detail & Related papers  (2025-06-03T01:37:09Z)
• Neural Antidote: Class-Wise Prompt Tuning for Purifying Backdoors in CLIP [51.04452017089568]
  Class-wise Backdoor Prompt Tuning (CBPT) is an efficient and effective defense mechanism that operates on text prompts to indirectly purify CLIP.<n>CBPT significantly mitigates backdoor threats while preserving model utility.
  arXiv  Detail & Related papers  (2025-02-26T16:25:15Z)
• Learn from the Past: A Proxy Guided Adversarial Defense Framework with Self Distillation Regularization [53.04697800214848]
  Adversarial Training (AT) is pivotal in fortifying the robustness of deep learning models. AT methods, relying on direct iterative updates for target model's defense, frequently encounter obstacles such as unstable training and catastrophic overfitting. We present a general proxy guided defense framework, LAST' (bf Learn from the Pbf ast)
  arXiv  Detail & Related papers  (2023-10-19T13:13:41Z)
• Avoid Adversarial Adaption in Federated Learning by Multi-Metric Investigations [55.2480439325792]
  Federated Learning (FL) facilitates decentralized machine learning model training, preserving data privacy, lowering communication costs, and boosting model performance through diversified data sources. FL faces vulnerabilities such as poisoning attacks, undermining model integrity with both untargeted performance degradation and targeted backdoor attacks. We define a new notion of strong adaptive adversaries, capable of adapting to multiple objectives simultaneously. MESAS is the first defense robust against strong adaptive adversaries, effective in real-world data scenarios, with an average overhead of just 24.37 seconds.
  arXiv  Detail & Related papers  (2023-06-06T11:44:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.