GPT-Red: Automated Red Teaming via Self-Play at Scale

• URL: http://arxiv.org/abs/2607.26115v1
• Date: Tue, 28 Jul 2026 16:03:39 GMT
• Title: GPT-Red: Automated Red Teaming via Self-Play at Scale
• Abstract summary: We introduce textbfGPT-Red, an automated red-teaming agent that is trained to discover novel prompt injection attacks against frontier LLMs.<n> GPT-Red excels at red-teaming: it reliably breaks our past models up to GPT-5.5, it finds more successful attacks than human red-teamers, and it generalizes to held-out environments.
• Score: 58.27696932324196
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce \textbf{GPT-Red}, an automated red-teaming agent that is trained to discover novel prompt injection attacks against frontier LLMs. The goal of this model is to evaluate and improve the robustness of our production systems. To this end, we use it to adversarially train GPT-5.6, our most robust model to prompt injections to date. To create GPT-Red, we design a scalable self-play algorithm where the model is tasked with attacking a diverse population of simultaneously-trained defender agents. We train the model on realistic red-teaming environments using compute on the same scale as some of our largest RL post-training runs, making it the single-largest LLM safety training run ever documented. GPT-Red excels at red-teaming: it reliably breaks our past models up to GPT-5.5, it finds more successful attacks than human red-teamers, and it generalizes to held-out environments, defender models, and harnesses. In the future, we expect that as we improve the robustness of each new GPT model, it will in turn will provide better learning signal for \textit{even stronger} red-teamer agents, thus unlocking a self-improvement flywheel.

Related papers

• Learning to Attack and Defend: Adaptive Red Teaming of Language Models via GRPO [2.714570887733442]
  Adv GRPO is a co-training framework that makes attacker-defender optimization using dense multi-channel rewards.<n>We show that our method can produce highly effective and transferable attacks and that co-trained defenders outperform baselines on safety benchmarks.
  arXiv  Detail & Related papers  (2026-06-08T16:21:36Z)
• Training a General Purpose Automated Red Teaming Model [11.120434965095555]
  We propose a pipeline for training a red teaming model that can generalize to arbitrary adversarial goals.<n>We demonstrate that finetuning small models, such as Qwen3-8B, using this pipeline results in a substantial improvement in their ability to generate attacks.
  arXiv  Detail & Related papers  (2026-04-24T23:37:17Z)
• AutoRed: A Free-form Adversarial Prompt Generation Framework for Automated Red Teaming [58.70941433155648]
  AutoRed is a free-form adversarial prompt generation framework that removes the need for seed instructions.<n>We build two red teaming datasets and evaluate eight state-of-the-art Large Language Models.<n>Our results highlight the limitations of seed-based approaches and demonstrate the potential of free-form red teaming for safety evaluation.
  arXiv  Detail & Related papers  (2025-10-09T15:17:28Z)
• RL Is a Hammer and LLMs Are Nails: A Simple Reinforcement Learning Recipe for Strong Prompt Injection [82.41836544860833]
  We introduce RL-Hammer, a simple recipe for training attacker models that automatically learn to perform strong prompt injections.<n>We propose a set of practical techniques that enable highly effective, universal attacks.<n> RL-Hammer reaches a 98% ASR against GPT-4o and a $72%$ ASR against GPT-5 with the Instruction Hierarchy defense.
  arXiv  Detail & Related papers  (2025-10-06T15:06:04Z)
• CoP: Agentic Red-teaming for Large Language Models using Composition of Principles [68.73212422583548]
  This paper proposes an agentic workflow to automate and scale the red-teaming process of Large Language Models (LLMs)<n>Human users provide a set of red-teaming principles as instructions to an AI agent to automatically orchestrate effective red-teaming strategies and generate jailbreak prompts.<n>When tested against leading LLMs, CoP reveals unprecedented safety risks by finding novel jailbreak prompts and improving the best-known single-turn attack success rate by up to 19.0 times.
  arXiv  Detail & Related papers  (2025-06-01T02:18:41Z)
• Capability-Based Scaling Laws for LLM Red-Teaming [71.89259138609965]
  Traditional prompt-engineering approaches may prove ineffective once red-teaming turns into a weak-to-strong problem.<n>We evaluate more than 500 attacker-target pairs using LLM-based jailbreak attacks that mimic human red-teamers.<n>We derive a jailbreaking scaling law that predicts attack success for a fixed target based on attacker-target capability gap.
  arXiv  Detail & Related papers  (2025-05-26T16:05:41Z)
• AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integration [40.350632196772466]
  This paper introduces AutoRedTeamer, a novel framework for fully automated, end-to-end red teaming against large language models (LLMs)<n>AutoRedTeamer combines a multi-agent architecture with a memory-guided attack selection mechanism to enable continuous discovery and integration of new attack vectors.<n>We demonstrate AutoRedTeamer's effectiveness across diverse evaluation settings, achieving 20% higher attack success rates on HarmBench against Llama-3.1-70B.
  arXiv  Detail & Related papers  (2025-03-20T00:13:04Z)
• Learning diverse attacks on large language models for robust red-teaming and safety tuning [126.32539952157083]
  Red-teaming, or identifying prompts that elicit harmful responses, is a critical step in ensuring the safe deployment of large language models.<n>We show that even with explicit regularization to favor novelty and diversity, existing approaches suffer from mode collapse or fail to generate effective attacks.<n>We propose to use GFlowNet fine-tuning, followed by a secondary smoothing phase, to train the attacker model to generate diverse and effective attack prompts.
  arXiv  Detail & Related papers  (2024-05-28T19:16:17Z)
• Evolving Diverse Red-team Language Models in Multi-round Multi-agent Games [11.873513881458747]
  Red team can identify vulnerabilities by attacking Large Language Model (LLM) to attain safety. Current efforts heavily rely on single-round prompt designs and unilateral red team optimizations against fixed blue teams. Here we introduce dynamic Red Team Game (RTG) to analyze the multi-round offensive and defensive interactions between red team and blue team.
  arXiv  Detail & Related papers  (2023-09-30T09:35:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.