PersonaTeaming: Exploring How Introducing Personas Can Improve Automated AI Red-Teaming

• URL: http://arxiv.org/abs/2509.03728v3
• Date: Mon, 27 Oct 2025 04:01:19 GMT
• Title: PersonaTeaming: Exploring How Introducing Personas Can Improve Automated AI Red-Teaming
• Abstract summary: PersonaTeaming is an initial step towards incorporating people's background and identities in automated red-teaming.<n>We develop a dynamic persona-generating algorithm that automatically generates various persona types adaptive to different seed prompts.<n>Experiments show promising improvements (up to 144.1%) in the attack success rates of adversarial prompts through persona mutation.
• Score: 22.601361185100235
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent developments in AI governance and safety research have called for red-teaming methods that can effectively surface potential risks posed by AI models. Many of these calls have emphasized how the identities and backgrounds of red-teamers can shape their red-teaming strategies, and thus the kinds of risks they are likely to uncover. While automated red-teaming approaches promise to complement human red-teaming by enabling larger-scale exploration of model behavior, current approaches do not consider the role of identity. As an initial step towards incorporating people's background and identities in automated red-teaming, we develop and evaluate a novel method, PersonaTeaming, that introduces personas in the adversarial prompt generation process to explore a wider spectrum of adversarial strategies. In particular, we first introduce a methodology for mutating prompts based on either "red-teaming expert" personas or "regular AI user" personas. We then develop a dynamic persona-generating algorithm that automatically generates various persona types adaptive to different seed prompts. In addition, we develop a set of new metrics to explicitly measure the "mutation distance" to complement existing diversity measurements of adversarial prompts. Our experiments show promising improvements (up to 144.1%) in the attack success rates of adversarial prompts through persona mutation, while maintaining prompt diversity, compared to RainbowPlus, a state-of-the-art automated red-teaming method. We discuss the strengths and limitations of different persona types and mutation methods, shedding light on future opportunities to explore complementarities between automated and human red-teaming approaches.

Related papers

• PersonaTeaming: Supporting Persona-Driven Red-Teaming for Generative AI [28.268811472721996]
  We develop PersonaTeaming, which incorporates personas into the adversarial prompt generation process.<n>We then instantiate PersonaTeaming as PersonaTeaming Playground, a user-facing interface that enables red-teamers to author their own personas and collaborate with AI to mutate and refine prompts.<n>In a user study with 11 industry practitioners, we found that PersonaTeaming Playground enabled diverse red-teaming strategies and outputs that practitioners perceived as useful.
  arXiv  Detail & Related papers  (2026-05-07T05:19:51Z)
• Cooperation Through Indirect Reciprocity in Child-Robot Interactions [81.62347137438248]
  We investigate whether indirect reciprocity can be transposed to children-robot interactions.<n>We find that IR extends to children and robots solving coordination dilemmas.<n>We observe that cooperating through multi-armed bandit algorithms is highly dependent on the strategies revealed by humans.
  arXiv  Detail & Related papers  (2025-11-07T07:08:32Z)
• AutoRed: A Free-form Adversarial Prompt Generation Framework for Automated Red Teaming [58.70941433155648]
  AutoRed is a free-form adversarial prompt generation framework that removes the need for seed instructions.<n>We build two red teaming datasets and evaluate eight state-of-the-art Large Language Models.<n>Our results highlight the limitations of seed-based approaches and demonstrate the potential of free-form red teaming for safety evaluation.
  arXiv  Detail & Related papers  (2025-10-09T15:17:28Z)
• Tree-based Dialogue Reinforced Policy Optimization for Red-Teaming Attacks [63.803415430308114]
  Current large language models are vulnerable to adversarial attacks in multi-turn interaction settings.<n>We propose DialTree-RPO, an on-policy reinforcement learning framework integrated with tree search.<n>Our approach achieves more than 25.9% higher ASR across 10 target models compared to previous state-of-the-art approaches.
  arXiv  Detail & Related papers  (2025-10-02T17:57:05Z)
• Dark Patterns Meet GUI Agents: LLM Agent Susceptibility to Manipulative Interfaces and the Role of Human Oversight [51.53020962098759]
  This study examines how agents, human participants, and human-AI teams respond to 16 types of dark patterns across diverse scenarios.<n>Phase 1 highlights that agents often fail to recognize dark patterns, and even when aware, prioritize task completion over protective action.<n>Phase 2 revealed divergent failure modes: humans succumb due to cognitive shortcuts and habitual compliance, while agents falter from procedural blind spots.
  arXiv  Detail & Related papers  (2025-09-12T22:26:31Z)
• Automatic LLM Red Teaming [18.044879441434432]
  We propose a novel paradigm: training an AI to strategically break' another AI.<n>Our generative agent learns coherent, multi-turn attack strategies through a fine-grained, token-level harm reward.<n>This approach sets a new state-of-the-art, fundamentally reframing red teaming as a dynamic, trajectory-based process.
  arXiv  Detail & Related papers  (2025-08-06T13:52:00Z)
• Red Teaming AI Red Teaming [9.942581294959107]
  We argue that there exists a significant gap between red teaming's original intent and its narrow focus on discovering model-level flaws in the context of generative AI.<n>We propose a comprehensive framework operationalizing red teaming in AI systems at two levels: macro-level system red teaming and micro-level model red teaming.
  arXiv  Detail & Related papers  (2025-07-07T23:23:40Z)
• Be a Multitude to Itself: A Prompt Evolution Framework for Red Teaming [24.303849271284676]
  Red teaming aims to find prompts that could elicit harmful responses from Large Language Models (LLMs)<n>We propose RTPE, a scalable evolution framework to evolve red teaming prompts across both breadth and depth dimensions.<n>Extensive experiments demonstrate that RTPE surpasses existing representative automatic red teaming methods on both attack success rate and diversity.
  arXiv  Detail & Related papers  (2025-02-22T06:13:19Z)
• Automated Red Teaming with GOAT: the Generative Offensive Agent Tester [8.947465706080523]
  Red teaming assesses how large language models can produce content that violates norms, policies, and rules set during their safety training. Most existing automated methods in the literature are not representative of the way humans tend to interact with AI models. We introduce Generative Offensive Agent Tester (GOAT), an automated agentic red teaming system that simulates plain language adversarial conversations.
  arXiv  Detail & Related papers  (2024-10-02T14:47:05Z)
• Holistic Automated Red Teaming for Large Language Models through Top-Down Test Case Generation and Multi-turn Interaction [24.499874512829198]
  We proposeHolistic Automated Red teaMing, which scales up the diversity of test cases based on an adversarial, fine-grained risk taxonomy. Our method also leverages a novel fine-tuning strategy and reinforcement learning techniques to facilitate multi-turn probing in a human-like manner.
  arXiv  Detail & Related papers  (2024-09-25T09:44:48Z)
• Learning diverse attacks on large language models for robust red-teaming and safety tuning [126.32539952157083]
  Red-teaming, or identifying prompts that elicit harmful responses, is a critical step in ensuring the safe deployment of large language models.<n>We show that even with explicit regularization to favor novelty and diversity, existing approaches suffer from mode collapse or fail to generate effective attacks.<n>We propose to use GFlowNet fine-tuning, followed by a secondary smoothing phase, to train the attacker model to generate diverse and effective attack prompts.
  arXiv  Detail & Related papers  (2024-05-28T19:16:17Z)
• Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts [57.49685172971446]
  We present Rainbow Teaming, a novel black-box approach for producing a diverse collection of adversarial prompts.<n>Our approach reveals hundreds of effective adversarial prompts, with an attack success rate exceeding 90%.<n>We additionally explore the versatility of Rainbow Teaming by applying it to question answering and cybersecurity.
  arXiv  Detail & Related papers  (2024-02-26T18:47:27Z)
• Red-Teaming for Generative AI: Silver Bullet or Security Theater? [42.35800543892003]
  We argue that while red-teaming may be a valuable big-tent idea for characterizing GenAI harm mitigations, industry may effectively apply red-teaming and other strategies behind closed doors to safeguard AI. To move toward a more robust toolbox of evaluations for generative AI, we synthesize our recommendations into a question bank meant to guide and scaffold future AI red-teaming practices.
  arXiv  Detail & Related papers  (2024-01-29T05:46:14Z)
• The Feasibility and Inevitability of Stealth Attacks [63.14766152741211]
  We study new adversarial perturbations that enable an attacker to gain control over decisions in generic Artificial Intelligence systems. In contrast to adversarial data modification, the attack mechanism we consider here involves alterations to the AI system itself.
  arXiv  Detail & Related papers  (2021-06-26T10:50:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.