Learning to Attack and Defend: Adaptive Red Teaming of Language Models via GRPO

• URL: http://arxiv.org/abs/2606.09701v1
• Date: Mon, 08 Jun 2026 16:21:36 GMT
• Title: Learning to Attack and Defend: Adaptive Red Teaming of Language Models via GRPO
• Abstract summary: Adv GRPO is a co-training framework that makes attacker-defender optimization using dense multi-channel rewards.<n>We show that our method can produce highly effective and transferable attacks and that co-trained defenders outperform baselines on safety benchmarks.
• Score: 2.714570887733442
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: AI red teaming must continually adapt to evolving attackers and defenders. Reinforcement learning offers a promising approach to discovering novel attacks, and co-training methods can produce more robust defenders in tandem. Recent works have demonstrated the efficacy of attacker-defender co-training by applying PPO and DPO, but report that GRPO is unstable in this setting. We introduce AdvGRPO, a co-training framework that makes GRPO viable for joint attacker-defender optimization using dense multi-channel rewards and decoupled advantage normalization. Training progresses through a curriculum from single-turn to closed-loop multi-turn attacks before bootstrapping co-training, where attacker and defender models are updated in alternation. We show that our method can produce highly effective and transferable attacks and that co-trained defenders outperform baselines on safety benchmarks.

Related papers

• CoER: Defending against Adaptive Indirect Prompt Injection via Adversarial Co-Evolution and Refinement [19.128257554975104]
  Language-model agents are vulnerable to indirect prompt injection (IPI) during tool use.<n>We propose CoER, a verifier-grounded co-evolution and refinement framework.<n>In our main seven-domain evaluation, CoER reduces observed overall attack success from 38.5% to 0.2%.
  arXiv  Detail & Related papers  (2026-09-07T14:09:11Z)
• Defending against Adaptive Prompt Injection Attacks via Reasoning-enabled Task Alignment [25.752599132396437]
  Indirect prompt injection attacks hijack LLM-based agents by embedding malicious instructions in third-party data that the agent retrieves during task execution.<n>Existing defenses report near-zero attack success rate on static benchmarks, yet recent adaptive evaluations show that these results collapse once the attacker is allowed to optimize against the deployed defense.<n>We propose RETA, a training-based method that grounds defense decisions on the user tasks rather than attacker-controlled data.
  arXiv  Detail & Related papers  (2026-06-13T19:15:44Z)
• CHASE: Adversarial Red-Blue Teaming for Improving LLM Safety using Reinforcement Learning [11.739543857396775]
  We introduce CHASE, a teaming framework in which a black-box attacker and a safety-aligned defender co-evolve.<n> CHASE cuts mean StrongREJECT score by 43.2% with 0% false-refusal on benign prompts.<n>Beyond the headline result, CHASE shows that template-free RL exploration recovers latent attack primitives that transfer across mechanistically distinct attack families.
  arXiv  Detail & Related papers  (2026-06-04T00:06:13Z)
• Hail to the Thief: Exploring Attacks and Defenses in Decentralised GRPO [0.9048611509540079]
  We present the first adversarial attack in decentralised GRPO.<n>We show that adversarial attacks can easily poison the benign nodes post-training, achieving attack success rates up to 100% in as few as 50 iterations.<n>We propose two ways to defend against these attacks, depending on whether all users train the same model or different models.
  arXiv  Detail & Related papers  (2025-11-12T22:29:07Z)
• Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Models [64.47869632167284]
  Conventional language model (LM) safety alignment relies on a reactive, disjoint procedure: attackers exploit a static model, followed by defensive fine-tuning to patch exposed vulnerabilities.<n>This sequential approach creates a mismatch -- attackers overfit to obsolete defenses, while defenders perpetually lag behind emerging threats.<n>We propose Self-RedTeam, an online self-play reinforcement learning algorithm where an attacker and defender agent co-evolve through continuous interaction.
  arXiv  Detail & Related papers  (2025-06-09T06:35:12Z)
• ARBoids: Adaptive Residual Reinforcement Learning With Boids Model for Cooperative Multi-USV Target Defense [0.918715978278858]
  This letter introduces ARBoids, a novel adaptive residual reinforcement learning framework.<n>It integrates deep reinforcement learning with the biologically inspired, force-based Boids model.<n>The proposed approach is validated in a high-fidelity Gazebo simulation environment.
  arXiv  Detail & Related papers  (2025-02-25T16:05:33Z)
• Adapting to Evolving Adversaries with Regularized Continual Robust Training [47.93633573641843]
  We present theoretical results which show that the gap in a model's robustness against different attacks is bounded by how far each attack perturbs a sample in the model's logit space.<n>Our findings and open-source code lay the groundwork for the deployment of models robust to evolving attacks.
  arXiv  Detail & Related papers  (2025-02-06T17:38:41Z)
• Learning diverse attacks on large language models for robust red-teaming and safety tuning [126.32539952157083]
  Red-teaming, or identifying prompts that elicit harmful responses, is a critical step in ensuring the safe deployment of large language models.<n>We show that even with explicit regularization to favor novelty and diversity, existing approaches suffer from mode collapse or fail to generate effective attacks.<n>We propose to use GFlowNet fine-tuning, followed by a secondary smoothing phase, to train the attacker model to generate diverse and effective attack prompts.
  arXiv  Detail & Related papers  (2024-05-28T19:16:17Z)
• Securing Recommender System via Cooperative Training [78.97620275467733]
  We propose a general framework, Triple Cooperative Defense (TCD), which employs three cooperative models that mutually enhance data. Considering existing attacks struggle to balance bi-level optimization and efficiency, we revisit poisoning attacks in recommender systems. We put forth a Game-based Co-training Attack (GCoAttack), which frames the proposed CoAttack and TCD as a game-theoretic process.
  arXiv  Detail & Related papers  (2024-01-23T12:07:20Z)
• Learning to Backdoor Federated Learning [9.046972927978997]
  In a federated learning (FL) system, malicious participants can easily embed backdoors into the aggregated model. We propose a general reinforcement learning-based backdoor attack framework. Our framework is both adaptive and flexible and achieves strong attack performance and durability even under state-of-the-art defenses.
  arXiv  Detail & Related papers  (2023-03-06T17:47:04Z)
• Guidance Through Surrogate: Towards a Generic Diagnostic Attack [101.36906370355435]
  We develop a guided mechanism to avoid local minima during attack optimization, leading to a novel attack dubbed Guided Projected Gradient Attack (G-PGA) Our modified attack does not require random restarts, large number of attack iterations or search for an optimal step-size. More than an effective attack, G-PGA can be used as a diagnostic tool to reveal elusive robustness due to gradient masking in adversarial defenses.
  arXiv  Detail & Related papers  (2022-12-30T18:45:23Z)
• Guided Adversarial Attack for Evaluating and Enhancing Adversarial Defenses [59.58128343334556]
  We introduce a relaxation term to the standard loss, that finds more suitable gradient-directions, increases attack efficacy and leads to more efficient adversarial training. We propose Guided Adversarial Margin Attack (GAMA), which utilizes function mapping of the clean image to guide the generation of adversaries. We also propose Guided Adversarial Training (GAT), which achieves state-of-the-art performance amongst single-step defenses.
  arXiv  Detail & Related papers  (2020-11-30T16:39:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.