AutoRed: A Free-form Adversarial Prompt Generation Framework for Automated Red Teaming

• URL: http://arxiv.org/abs/2510.08329v1
• Date: Thu, 09 Oct 2025 15:17:28 GMT
• Title: AutoRed: A Free-form Adversarial Prompt Generation Framework for Automated Red Teaming
• Authors: Muxi Diao, Yutao Mou, Keqing He, Hanbo Song, Lulu Zhao, Shikun Zhang, Wei Ye, Kongming Liang, Zhanyu Ma,
• Abstract summary: AutoRed is a free-form adversarial prompt generation framework that removes the need for seed instructions.<n>We build two red teaming datasets and evaluate eight state-of-the-art Large Language Models.<n>Our results highlight the limitations of seed-based approaches and demonstrate the potential of free-form red teaming for safety evaluation.
• Score: 58.70941433155648
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The safety of Large Language Models (LLMs) is crucial for the development of trustworthy AI applications. Existing red teaming methods often rely on seed instructions, which limits the semantic diversity of the synthesized adversarial prompts. We propose AutoRed, a free-form adversarial prompt generation framework that removes the need for seed instructions. AutoRed operates in two stages: (1) persona-guided adversarial instruction generation, and (2) a reflection loop to iteratively refine low-quality prompts. To improve efficiency, we introduce a verifier to assess prompt harmfulness without querying the target models. Using AutoRed, we build two red teaming datasets -- AutoRed-Medium and AutoRed-Hard -- and evaluate eight state-of-the-art LLMs. AutoRed achieves higher attack success rates and better generalization than existing baselines. Our results highlight the limitations of seed-based approaches and demonstrate the potential of free-form red teaming for LLM safety evaluation. We will open source our datasets in the near future.

Related papers

• Automatic LLM Red Teaming [18.044879441434432]
  We propose a novel paradigm: training an AI to strategically break' another AI.<n>Our generative agent learns coherent, multi-turn attack strategies through a fine-grained, token-level harm reward.<n>This approach sets a new state-of-the-art, fundamentally reframing red teaming as a dynamic, trajectory-based process.
  arXiv  Detail & Related papers  (2025-08-06T13:52:00Z)
• CoP: Agentic Red-teaming for Large Language Models using Composition of Principles [61.404771120828244]
  This paper proposes an agentic workflow to automate and scale the red-teaming process of Large Language Models (LLMs)<n>Human users provide a set of red-teaming principles as instructions to an AI agent to automatically orchestrate effective red-teaming strategies and generate jailbreak prompts.<n>When tested against leading LLMs, CoP reveals unprecedented safety risks by finding novel jailbreak prompts and improving the best-known single-turn attack success rate by up to 19.0 times.
  arXiv  Detail & Related papers  (2025-06-01T02:18:41Z)
• AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integration [40.350632196772466]
  This paper introduces AutoRedTeamer, a novel framework for fully automated, end-to-end red teaming against large language models (LLMs)<n>AutoRedTeamer combines a multi-agent architecture with a memory-guided attack selection mechanism to enable continuous discovery and integration of new attack vectors.<n>We demonstrate AutoRedTeamer's effectiveness across diverse evaluation settings, achieving 20% higher attack success rates on HarmBench against Llama-3.1-70B.
  arXiv  Detail & Related papers  (2025-03-20T00:13:04Z)
• Be a Multitude to Itself: A Prompt Evolution Framework for Red Teaming [24.303849271284676]
  Red teaming aims to find prompts that could elicit harmful responses from Large Language Models (LLMs)<n>We propose RTPE, a scalable evolution framework to evolve red teaming prompts across both breadth and depth dimensions.<n>Extensive experiments demonstrate that RTPE surpasses existing representative automatic red teaming methods on both attack success rate and diversity.
  arXiv  Detail & Related papers  (2025-02-22T06:13:19Z)
• Text-Diffusion Red-Teaming of Large Language Models: Unveiling Harmful Behaviors with Proximity Constraints [20.542545906686318]
  We introduce a black-box red-teaming method inspired by text-diffusion models: Diffusion for Auditing and Red-Teaming (DART)<n>DART modifies the reference prompt by perturbing it in the embedding space, directly controlling the amount of change introduced.<n>Our results show that DART is significantly more effective at discovering harmful inputs in close proximity to the reference prompt.
  arXiv  Detail & Related papers  (2025-01-14T16:32:01Z)
• Gradient-Based Language Model Red Teaming [9.972783485792885]
  Red teaming is a strategy for identifying weaknesses in generative language models (LMs) Red teaming is instrumental for both model alignment and evaluation, but is labor-intensive and difficult to scale when done by humans. We present Gradient-Based Red Teaming (GBRT), a red teaming method for automatically generating diverse prompts that are likely to cause an LM to output unsafe responses.
  arXiv  Detail & Related papers  (2024-01-30T01:19:25Z)
• MART: Improving LLM Safety with Multi-round Automatic Red-Teaming [72.2127916030909]
  We propose a Multi-round Automatic Red-Teaming (MART) method, which incorporates both automatic adversarial prompt writing and safe response generation. On adversarial prompt benchmarks, the violation rate of an LLM with limited safety alignment reduces up to 84.7% after 4 rounds of MART. Notably, model helpfulness on non-adversarial prompts remains stable throughout iterations, indicating the target LLM maintains strong performance on instruction following.
  arXiv  Detail & Related papers  (2023-11-13T19:13:29Z)
• AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models [55.748851471119906]
  Safety alignment of Large Language Models (LLMs) can be compromised with manual jailbreak attacks and (automatic) adversarial attacks. Recent studies suggest that defending against these attacks is possible: adversarial attacks generate unlimited but unreadable gibberish prompts, detectable by perplexity-based filters. We introduce AutoDAN, an interpretable, gradient-based adversarial attack that merges the strengths of both attack types.
  arXiv  Detail & Related papers  (2023-10-23T17:46:07Z)
• Attack Prompt Generation for Red Teaming and Defending Large Language Models [70.157691818224]
  Large language models (LLMs) are susceptible to red teaming attacks, which can induce LLMs to generate harmful content. We propose an integrated approach that combines manual and automatic methods to economically generate high-quality attack prompts.
  arXiv  Detail & Related papers  (2023-10-19T06:15:05Z)
• FLIRT: Feedback Loop In-context Red Teaming [79.63896510559357]
  We propose an automatic red teaming framework that evaluates a given black-box model and exposes its vulnerabilities. Our framework uses in-context learning in a feedback loop to red team models and trigger them into unsafe content generation.
  arXiv  Detail & Related papers  (2023-08-08T14:03:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.