Adaptive Evaluation of Out-of-Band Defenses Against Prompt Injection in LLM Agents

• URL: http://arxiv.org/abs/2606.26479v1
• Date: Thu, 25 Jun 2026 00:35:23 GMT
• Title: Adaptive Evaluation of Out-of-Band Defenses Against Prompt Injection in LLM Agents
• Abstract summary: Recent work has converged on a strategy for defending tool-using LLM agents against indirect prompt injection.<n>We organize these out-of-band defenses as instances of classical integrity protection.<n>We run that protocol as an independent reproduction and extension of Progent's own adaptive-attack analysis.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent work (2024 to 2026) has converged on a strategy for defending tool-using LLM agents against indirect prompt injection: rather than training the model to refuse malicious instructions, enforce security outside the model with a deterministic policy that mediates the agent's actions. Systems such as CaMeL, FIDES, Progent, RTBAS, and FORGE realize this with capabilities, information-flow labels, and reference monitors, and several report near-elimination of attacks on the AgentDojo benchmark. We make two contributions. First, we organize these out-of-band defenses as instances of classical integrity protection (Biba), reference monitoring, and least privilege, yielding a structured comparison of what they do and do not cover. Second, we warn that every one of them is validated only on static benchmarks (a fixed set of injection attempts), the same methodology that made in-band defenses look strong until adaptive, defense-aware attacks broke twelve of them at over 90% success; we specify the threat model and protocol an adaptive evaluation requires. We then run that protocol as an independent reproduction and extension of Progent's own adaptive-attack analysis, on AgentDojo, with an open-weight agent (Qwen2.5-7B) self-hosted on a single H200, a setting its authors did not test. Averaged over three runs, the defense held: Progent cut mean attack success roughly sixfold (25.8% to 4.2%), and a hand-crafted adaptive attack did not raise it (2.6%). This is one small-scale data point on a weak model with a single black-box attack template; a stronger optimized (white-box GCG) attack remains open. The result is consistent with, but does not establish, the hypothesis that deterministic out-of-band enforcement is a harder target for an adaptive attacker than in-band detection.

Related papers

• Toward Metacognitive One-Shot Indirect Prompt Injection: Strategy Abstraction Via Outcome-Conditioned Reflection [17.713444943152506]
  Tool-using large language model (LLM) agents are vulnerable to indirect prompt injection (IPI)<n>Most existing adaptive attacks rely on repeatedly querying and refining against the target agent.<n>We propose SAVOR, which shifts attack adaptation from test-time to offline strategy distillation.
  arXiv  Detail & Related papers  (2026-08-09T16:19:05Z)
• Defending against Adaptive Prompt Injection Attacks via Reasoning-enabled Task Alignment [25.752599132396437]
  Indirect prompt injection attacks hijack LLM-based agents by embedding malicious instructions in third-party data that the agent retrieves during task execution.<n>Existing defenses report near-zero attack success rate on static benchmarks, yet recent adaptive evaluations show that these results collapse once the attacker is allowed to optimize against the deployed defense.<n>We propose RETA, a training-based method that grounds defense decisions on the user tasks rather than attacker-controlled data.
  arXiv  Detail & Related papers  (2026-06-13T19:15:44Z)
• AutoDojo: Adaptive Black-Box Attacks Reveal the Limits of IPI Defenses and Task-Specification Effects in LLM Agents [57.34566159148893]
  Indirect prompt injection (IPI) is a major security threat to LLM-powered agents.<n>We develop AutoDojo, an adaptive extension of AgentDojo that optimize IPI against a given defense.
  arXiv  Detail & Related papers  (2026-06-13T02:09:08Z)
• PlanGuard: Defending Agents against Indirect Prompt Injection via Planning-based Consistency Verification [1.7904458681854372]
  PlanGuard is a training-free defense framework based on the principle of Context Isolation.<n>PlanGuard effectively neutralizes attacks, reducing the Attack Success Rate (ASR) from 72.8% to 0%.
  arXiv  Detail & Related papers  (2026-04-11T09:59:46Z)
• ReasAlign: Reasoning Enhanced Safety Alignment against Prompt Injection Attack [52.17935054046577]
  We present ReasAlign, a model-level solution to improve safety alignment against indirect prompt injection attacks.<n>ReasAlign incorporates structured reasoning steps to analyze user queries, detect conflicting instructions, and preserve the continuity of the user's intended tasks.
  arXiv  Detail & Related papers  (2026-01-15T08:23:38Z)
• AI Security Beyond Core Domains: Resume Screening as a Case Study of Adversarial Vulnerabilities in Specialized LLM Applications [71.27518152526686]
  Large Language Models (LLMs) excel at text comprehension and generation, making them ideal for automated tasks like code review and content moderation.<n>LLMs can be manipulated by "adversarial instructions" hidden in input data, such as resumes or code, causing them to deviate from their intended task.<n>This paper introduces a benchmark to assess this vulnerability in resume screening, revealing attack success rates exceeding 80% for certain attack types.
  arXiv  Detail & Related papers  (2025-12-23T08:42:09Z)
• AegisLLM: Scaling Agentic Systems for Self-Reflective Defense in LLM Security [74.22452069013289]
  AegisLLM is a cooperative multi-agent defense against adversarial attacks and information leakage.<n>We show that scaling agentic reasoning system at test-time substantially enhances robustness without compromising model utility.<n> Comprehensive evaluations across key threat scenarios, including unlearning and jailbreaking, demonstrate the effectiveness of AegisLLM.
  arXiv  Detail & Related papers  (2025-04-29T17:36:05Z)
• Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents [3.5248694676821484]
  We evaluate eight different defenses and bypass all of them using adaptive attacks, consistently achieving an attack success rate of over 50%.<n>Our research underscores the need for adaptive attack evaluation when designing defenses to ensure robustness and reliability.
  arXiv  Detail & Related papers  (2025-02-27T04:04:50Z)
• Practical Evaluation of Adversarial Robustness via Adaptive Auto Attack [96.50202709922698]
  A practical evaluation method should be convenient (i.e., parameter-free), efficient (i.e., fewer iterations) and reliable. We propose a parameter-free Adaptive Auto Attack (A$3$) evaluation method which addresses the efficiency and reliability in a test-time-training fashion.
  arXiv  Detail & Related papers  (2022-03-10T04:53:54Z)
• Label-Only Membership Inference Attacks [67.46072950620247]
  We introduce label-only membership inference attacks. Our attacks evaluate the robustness of a model's predicted labels under perturbations. We find that training models with differential privacy and (strong) L2 regularization are the only known defense strategies.
  arXiv  Detail & Related papers  (2020-07-28T15:44:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.