CoER: Defending against Adaptive Indirect Prompt Injection via Adversarial Co-Evolution and Refinement

• URL: http://arxiv.org/abs/2609.07529v2
• Date: Tue, 15 Sep 2026 17:40:12 GMT
• Title: CoER: Defending against Adaptive Indirect Prompt Injection via Adversarial Co-Evolution and Refinement
• Abstract summary: Language-model agents are vulnerable to indirect prompt injection (IPI) during tool use.<n>We propose CoER, a verifier-grounded co-evolution and refinement framework.<n>In our main seven-domain evaluation, CoER reduces observed overall attack success from 38.5% to 0.2%.
• Score: 19.128257554975104
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Language-model agents are vulnerable to indirect prompt injection (IPI) during tool use: adversarial instructions hidden in untrusted tool outputs can covertly redirect legitimate task execution. Existing work often trains and evaluates defenses against fixed attacks that do not adapt to the defender's behavior, so the resulting defenses may struggle against adaptive attacks. We combine adaptive attacker-defender co-training with subsequent refinement: continued interaction improves both roles, while learned attackers provide training challenges for further gains in defender safety and task utility. We therefore model adaptive IPI as a general-sum Markov game: the defender advances the task through successive tool calls, while the attacker can inject multiple times within the same task and adapt subsequent attacks to the defender's responses. Building on this formulation, we propose CoER, a verifier-grounded co-evolution and refinement framework. After initializing the attacker from successful trajectories, Co-PPO retains historical policies from both roles as opponent populations and mixes current and historical opponents for bilateral reinforcement learning, extending training beyond the latest matchup. Attackers from these populations are then reused to challenge teacher agents, and only demonstrations verified for both safety and task completion are used to fine-tune the co-evolved defender. In our main seven-domain evaluation, CoER reduces observed overall attack success from 38.5% to 0.2% and raises task utility from 63.2% to 76.3%, with improved attack resistance on external benchmarks.

Related papers

• Toward Metacognitive One-Shot Indirect Prompt Injection: Strategy Abstraction Via Outcome-Conditioned Reflection [17.713444943152506]
  Tool-using large language model (LLM) agents are vulnerable to indirect prompt injection (IPI)<n>Most existing adaptive attacks rely on repeatedly querying and refining against the target agent.<n>We propose SAVOR, which shifts attack adaptation from test-time to offline strategy distillation.
  arXiv  Detail & Related papers  (2026-08-09T16:19:05Z)
• Stateful Cooperative Agents Safeguarding LLMs Against Evolving Multi-Turn Attacks [56.45762972211923]
  We present a proactive defense framework for securing LLMs against evolving multi-turn adversarial attacks.<n>It employs a cooperative multi-agent architecture in which specialized agents execute complementary defense strategies.<n>We present the EMRA dataset designed to simulate evolving strategies across multi-turn attacks.
  arXiv  Detail & Related papers  (2026-07-31T14:04:49Z)
• Defending against Adaptive Prompt Injection Attacks via Reasoning-enabled Task Alignment [25.752599132396437]
  Indirect prompt injection attacks hijack LLM-based agents by embedding malicious instructions in third-party data that the agent retrieves during task execution.<n>Existing defenses report near-zero attack success rate on static benchmarks, yet recent adaptive evaluations show that these results collapse once the attacker is allowed to optimize against the deployed defense.<n>We propose RETA, a training-based method that grounds defense decisions on the user tasks rather than attacker-controlled data.
  arXiv  Detail & Related papers  (2026-06-13T19:15:44Z)
• Learning to Attack and Defend: Adaptive Red Teaming of Language Models via GRPO [2.714570887733442]
  Adv GRPO is a co-training framework that makes attacker-defender optimization using dense multi-channel rewards.<n>We show that our method can produce highly effective and transferable attacks and that co-trained defenders outperform baselines on safety benchmarks.
  arXiv  Detail & Related papers  (2026-06-08T16:21:36Z)
• CHASE: Adversarial Red-Blue Teaming for Improving LLM Safety using Reinforcement Learning [11.739543857396775]
  We introduce CHASE, a teaming framework in which a black-box attacker and a safety-aligned defender co-evolve.<n> CHASE cuts mean StrongREJECT score by 43.2% with 0% false-refusal on benign prompts.<n>Beyond the headline result, CHASE shows that template-free RL exploration recovers latent attack primitives that transfer across mechanistically distinct attack families.
  arXiv  Detail & Related papers  (2026-06-04T00:06:13Z)
• CoopGuard: Stateful Cooperative Agents Safeguarding LLMs Against Evolving Multi-Round Attacks [31.582651893036683]
  CoopGuard is a stateful multi-round LLM defense framework based on cooperative agents.<n>It employs three specialized agents for complementary round-level strategies.<n> Experiments show that CoopGuard reduces attack success rate by 78.9% over state-of-the-art defenses.
  arXiv  Detail & Related papers  (2026-04-05T11:06:13Z)
• AdvEvo-MARL: Shaping Internalized Safety through Adversarial Co-Evolution in Multi-Agent Reinforcement Learning [78.5751183537704]
  AdvEvo-MARL is a co-evolutionary multi-agent reinforcement learning framework that internalizes safety into task agents.<n>Rather than relying on external guards, AdvEvo-MARL jointly optimize attackers and defenders.
  arXiv  Detail & Related papers  (2025-10-02T02:06:30Z)
• Hammer and Anvil: A Principled Defense Against Backdoors in Federated Learning [19.849567299082306]
  Federated Learning is a distributed learning technique in which multiple clients cooperate to train a machine learning model.<n>In this work, we first devise a new adaptive adversary that surpasses existing adversaries in capabilities.<n>Then, we present Hammer and Anvil, a principled defense approach that combines two defenses in their underlying principle.
  arXiv  Detail & Related papers  (2025-09-09T18:54:31Z)
• Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Models [64.47869632167284]
  Conventional language model (LM) safety alignment relies on a reactive, disjoint procedure: attackers exploit a static model, followed by defensive fine-tuning to patch exposed vulnerabilities.<n>This sequential approach creates a mismatch -- attackers overfit to obsolete defenses, while defenders perpetually lag behind emerging threats.<n>We propose Self-RedTeam, an online self-play reinforcement learning algorithm where an attacker and defender agent co-evolve through continuous interaction.
  arXiv  Detail & Related papers  (2025-06-09T06:35:12Z)
• On the Difficulty of Defending Contrastive Learning against Backdoor Attacks [58.824074124014224]
  We show how contrastive backdoor attacks operate through distinctive mechanisms. Our findings highlight the need for defenses tailored to the specificities of contrastive backdoor attacks.
  arXiv  Detail & Related papers  (2023-12-14T15:54:52Z)
• Improving the Adversarial Robustness for Speaker Verification by Self-Supervised Learning [95.60856995067083]
  This work is among the first to perform adversarial defense for ASV without knowing the specific attack algorithms. We propose to perform adversarial defense from two perspectives: 1) adversarial perturbation purification and 2) adversarial perturbation detection. Experimental results show that our detection module effectively shields the ASV by detecting adversarial samples with an accuracy of around 80%.
  arXiv  Detail & Related papers  (2021-06-01T07:10:54Z)
• What Doesn't Kill You Makes You Robust(er): Adversarial Training against Poisons and Backdoors [57.040948169155925]
  We extend the adversarial training framework to defend against (training-time) poisoning and backdoor attacks. Our method desensitizes networks to the effects of poisoning by creating poisons during training and injecting them into training batches. We show that this defense withstands adaptive attacks, generalizes to diverse threat models, and incurs a better performance trade-off than previous defenses.
  arXiv  Detail & Related papers  (2021-02-26T17:54:36Z)
• Guided Adversarial Attack for Evaluating and Enhancing Adversarial Defenses [59.58128343334556]
  We introduce a relaxation term to the standard loss, that finds more suitable gradient-directions, increases attack efficacy and leads to more efficient adversarial training. We propose Guided Adversarial Margin Attack (GAMA), which utilizes function mapping of the clean image to guide the generation of adversaries. We also propose Guided Adversarial Training (GAT), which achieves state-of-the-art performance amongst single-step defenses.
  arXiv  Detail & Related papers  (2020-11-30T16:39:39Z)
• On Adaptive Attacks to Adversarial Example Defenses [123.32678153377915]
  This paper lays out the methodology and the approach necessary to perform an adaptive attack against defenses to adversarial examples. We hope that these analyses will serve as guidance on how to properly perform adaptive attacks against defenses to adversarial examples.
  arXiv  Detail & Related papers  (2020-02-19T18:50:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.