Defending against Adaptive Prompt Injection Attacks via Reasoning-enabled Task Alignment

• URL: http://arxiv.org/abs/2606.15441v1
• Date: Sat, 13 Jun 2026 19:15:44 GMT
• Title: Defending against Adaptive Prompt Injection Attacks via Reasoning-enabled Task Alignment
• Abstract summary: Indirect prompt injection attacks hijack LLM-based agents by embedding malicious instructions in third-party data that the agent retrieves during task execution.<n>Existing defenses report near-zero attack success rate on static benchmarks, yet recent adaptive evaluations show that these results collapse once the attacker is allowed to optimize against the deployed defense.<n>We propose RETA, a training-based method that grounds defense decisions on the user tasks rather than attacker-controlled data.
• Score: 25.752599132396437
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Indirect prompt injection attacks hijack LLM-based agents by embedding malicious instructions in third-party data that the agent retrieves during task execution. Existing defenses report near-zero attack success rate on static benchmarks, yet recent adaptive evaluations show that these results collapse once the attacker is allowed to optimize against the deployed defense. In this work, we trace this collapse to two failure modes. First, existing defense methods are confined to recognizing specific attack patterns, rather than assessing whether the intent of every embedded instruction is relevant to the user task. Second, training-based defenses, which otherwise offer the strongest safety-utility trade-off, assemble their adversarial examples from a handful of hand-crafted templates, and the resulting defender fails to generalize outside that narrow strategy distribution. To address these gaps, we propose RETA, a training-based method that grounds defense decisions on the user tasks rather than attacker-controlled data. At each tool-output step, the defender undertakes chain-of-thought reasoning verifying that its actions are consistent with the user task. Leveraging red-teaming, a simulated attacker synthesizes adversarial training data and receives a dictionary-learning diversity reward, achieving broad coverage of injection-reformulation strategies. Together, these allow the defender to be optimized via multi-objective reinforcement learning and achieve better safety-utility trade-off. Across six black-box adaptive attacks, RETA keeps every per-attack ASR below 10%, with average ASR of 2.92% and 3.75% on the two target models, while preserving most utility under attack and on clean inputs.

Related papers

• CoER: Defending against Adaptive Indirect Prompt Injection via Adversarial Co-Evolution and Refinement [19.128257554975104]
  Language-model agents are vulnerable to indirect prompt injection (IPI) during tool use.<n>We propose CoER, a verifier-grounded co-evolution and refinement framework.<n>In our main seven-domain evaluation, CoER reduces observed overall attack success from 38.5% to 0.2%.
  arXiv  Detail & Related papers  (2026-09-07T14:09:11Z)
• Toward Metacognitive One-Shot Indirect Prompt Injection: Strategy Abstraction Via Outcome-Conditioned Reflection [17.713444943152506]
  Tool-using large language model (LLM) agents are vulnerable to indirect prompt injection (IPI)<n>Most existing adaptive attacks rely on repeatedly querying and refining against the target agent.<n>We propose SAVOR, which shifts attack adaptation from test-time to offline strategy distillation.
  arXiv  Detail & Related papers  (2026-08-09T16:19:05Z)
• Stateful Cooperative Agents Safeguarding LLMs Against Evolving Multi-Turn Attacks [56.45762972211923]
  We present a proactive defense framework for securing LLMs against evolving multi-turn adversarial attacks.<n>It employs a cooperative multi-agent architecture in which specialized agents execute complementary defense strategies.<n>We present the EMRA dataset designed to simulate evolving strategies across multi-turn attacks.
  arXiv  Detail & Related papers  (2026-07-31T14:04:49Z)
• Adaptive Evaluation of Out-of-Band Defenses Against Prompt Injection in LLM Agents [0.0]
  Recent work has converged on a strategy for defending tool-using LLM agents against indirect prompt injection.<n>We organize these out-of-band defenses as instances of classical integrity protection.<n>We run that protocol as an independent reproduction and extension of Progent's own adaptive-attack analysis.
  arXiv  Detail & Related papers  (2026-06-25T00:35:23Z)
• AutoDojo: Adaptive Attacks Expose Superficial Defenses and User-Underspecification Limits in LLM Agents [57.34566159148893]
  Indirect prompt injection (IPI) is a major security threat to LLM-powered agents.<n>We develop AutoDojo, an adaptive extension of AgentDojo that optimize IPI against a given defense.
  arXiv  Detail & Related papers  (2026-06-13T02:09:08Z)
• The Autonomy Tax: Defense Training Breaks LLM Agents [5.990318568221089]
  Defense training designed to improve safety systematically destroys agent competence while failing to prevent sophisticated attacks.<n>We evaluate defended models against undefended baselines across 97 agent tasks and 1,000 adversarial prompts.<n>Our findings demonstrate that current defense paradigms optimize for single-turn refusal benchmarks while rendering multi-step agents fundamentally unreliable.
  arXiv  Detail & Related papers  (2026-03-19T19:33:17Z)
• Dashed Line Defense: Plug-And-Play Defense Against Adaptive Score-Based Query Attacks [3.206339985805037]
  Dashed Line Defense (DLD) is a plug-and-play post-processing method specifically designed to withstand adaptive query strategies.<n>By introducing ambiguity in how the observed loss reflects the true adversarial strength of candidate examples, DLD prevents attackers from reliably analyzing and adapting their queries.<n>We provide theoretical guarantees of DLD's defense capability and validate its effectiveness through experiments on ImageNet.
  arXiv  Detail & Related papers  (2026-02-09T14:02:32Z)
• The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injections [74.60337113759313]
  Current defenses against jailbreaks and prompt injections are typically evaluated against a static set of harmful attack strings.<n>We argue that this evaluation process is flawed. Instead, we should evaluate defenses against adaptive attackers who explicitly modify their attack strategy to counter a defense's design.
  arXiv  Detail & Related papers  (2025-10-10T05:51:04Z)
• Adversarial Reinforcement Learning for Large Language Model Agent Safety [20.704989548285372]
  Large Language Model (LLM) agents can leverage tools like Google Search to complete complex tasks.<n>Current defense strategies rely on fine-tuning LLM agents on datasets of known attacks.<n>We propose Adversarial Reinforcement Learning for Agent Safety (ARLAS), a novel framework that leverages adversarial reinforcement learning (RL) by formulating the problem as a two-player zero-sum game.
  arXiv  Detail & Related papers  (2025-10-06T23:09:18Z)
• Benchmarking Misuse Mitigation Against Covert Adversaries [80.74502950627736]
  Existing language model safety evaluations focus on overt attacks and low-stakes tasks.<n>We develop Benchmarks for Stateful Defenses (BSD), a data generation pipeline that automates evaluations of covert attacks and corresponding defenses.<n>Our evaluations indicate that decomposition attacks are effective misuse enablers, and highlight stateful defenses as a countermeasure.
  arXiv  Detail & Related papers  (2025-06-06T17:33:33Z)
• Adversarial Inception Backdoor Attacks against Reinforcement Learning [16.350898218047405]
  Recent works have demonstrated the vulnerability of Deep Reinforcement Learning (DRL) algorithms against training-time, backdoor poisoning attacks.<n>We propose a new class of backdoor attacks against DRL which are the first to achieve state of the art performance under strict reward constraints.
  arXiv  Detail & Related papers  (2024-10-17T19:50:28Z)
• Improving the Adversarial Robustness for Speaker Verification by Self-Supervised Learning [95.60856995067083]
  This work is among the first to perform adversarial defense for ASV without knowing the specific attack algorithms. We propose to perform adversarial defense from two perspectives: 1) adversarial perturbation purification and 2) adversarial perturbation detection. Experimental results show that our detection module effectively shields the ASV by detecting adversarial samples with an accuracy of around 80%.
  arXiv  Detail & Related papers  (2021-06-01T07:10:54Z)
• Guided Adversarial Attack for Evaluating and Enhancing Adversarial Defenses [59.58128343334556]
  We introduce a relaxation term to the standard loss, that finds more suitable gradient-directions, increases attack efficacy and leads to more efficient adversarial training. We propose Guided Adversarial Margin Attack (GAMA), which utilizes function mapping of the clean image to guide the generation of adversaries. We also propose Guided Adversarial Training (GAT), which achieves state-of-the-art performance amongst single-step defenses.
  arXiv  Detail & Related papers  (2020-11-30T16:39:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.