AgenticRed: Optimizing Agentic Systems for Automated Red-teaming

• URL: http://arxiv.org/abs/2601.13518v1
• Date: Tue, 20 Jan 2026 02:10:22 GMT
• Title: AgenticRed: Optimizing Agentic Systems for Automated Red-teaming
• Authors: Jiayi Yuan, Jonathan Nöther, Natasha Jaques, Goran Radanović,
• Abstract summary: We introduce AgenticRed, an automated pipeline to iteratively design and refine red-teaming systems without human intervention.<n>Inspired by methods like Meta Agent Search, we develop a novel procedure for evolving agentic systems using evolutionary selection.<n>Red-teaming systems designed by AgenticRed consistently outperform state-of-the-art approaches.
• Score: 10.257924099620295
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While recent automated red-teaming methods show promise for systematically exposing model vulnerabilities, most existing approaches rely on human-specified workflows. This dependence on manually designed workflows suffers from human biases and makes exploring the broader design space expensive. We introduce AgenticRed, an automated pipeline that leverages LLMs' in-context learning to iteratively design and refine red-teaming systems without human intervention. Rather than optimizing attacker policies within predefined structures, AgenticRed treats red-teaming as a system design problem. Inspired by methods like Meta Agent Search, we develop a novel procedure for evolving agentic systems using evolutionary selection, and apply it to the problem of automatic red-teaming. Red-teaming systems designed by AgenticRed consistently outperform state-of-the-art approaches, achieving 96% attack success rate (ASR) on Llama-2-7B (36% improvement) and 98% on Llama-3-8B on HarmBench. Our approach exhibits strong transferability to proprietary models, achieving 100% ASR on GPT-3.5-Turbo and GPT-4o-mini, and 60% on Claude-Sonnet-3.5 (24% improvement). This work highlights automated system design as a powerful paradigm for AI safety evaluation that can keep pace with rapidly evolving models.

Related papers

• EmboCoach-Bench: Benchmarking AI Agents on Developing Embodied Robots [68.29056647487519]
  Embodied AI is fueled by high-fidelity simulation and large-scale data collection.<n>However, this scaling capability remains bottlenecked by a reliance on labor-intensive manual oversight.<n>We introduce textscEmboCoach-Bench, a benchmark evaluating the capacity of LLM agents to autonomously engineer embodied policies.
  arXiv  Detail & Related papers  (2026-01-29T11:33:49Z)
• ROAD: Reflective Optimization via Automated Debugging for Zero-Shot Agent Alignment [1.6968020497268546]
  ROAD is a novel framework that treats optimization as a dynamic debug investigation rather than a search.<n>Road is highly sample-efficient, achieving a 5.6 percent increase in success rate and a 3.8 percent increase in search accuracy.<n>These findings suggest that mimicking the human engineering loop of failure analysis and patching offers a viable, data-efficient alternative to resource-intensive training.
  arXiv  Detail & Related papers  (2025-12-30T07:31:34Z)
• SIRAJ: Diverse and Efficient Red-Teaming for LLM Agents via Distilled Structured Reasoning [18.219912912964812]
  We present SIRAJ: a generic red-teaming framework for arbitrary black-box LLM agents.<n>We employ a dynamic two-step process that starts with an agent definition and generates diverse seed test cases.<n>It iteratively constructs and refines model-based adversarial attacks based on the execution trajectories of former attempts.
  arXiv  Detail & Related papers  (2025-10-30T00:32:58Z)
• Introducing LongCat-Flash-Thinking: A Technical Report [116.75498493511026]
  LongCat-Flash-Thinking is an efficient open-source Mixture-of-Experts (MoE) reasoning model.<n>Its advanced capabilities are cultivated through a meticulously crafted training process.<n>LongCat-Flash-Thinking achieves state-of-the-art performance among open-source models on a suite of complex reasoning tasks.
  arXiv  Detail & Related papers  (2025-09-23T10:25:48Z)
• SwarmAgentic: Towards Fully Automated Agentic System Generation via Swarm Intelligence [28.042768995386037]
  We propose SwarmAgentic, a framework for fully automated agentic system generation.<n>SwarmAgentic constructs agentic systems from scratch and jointly optimize agent functionality and collaboration.<n>We evaluate our method on six real-world, open-ended, and exploratory tasks involving high-level planning, system-level coordination, and creative reasoning.
  arXiv  Detail & Related papers  (2025-06-18T17:54:55Z)
• CoP: Agentic Red-teaming for Large Language Models using Composition of Principles [68.73212422583548]
  This paper proposes an agentic workflow to automate and scale the red-teaming process of Large Language Models (LLMs)<n>Human users provide a set of red-teaming principles as instructions to an AI agent to automatically orchestrate effective red-teaming strategies and generate jailbreak prompts.<n>When tested against leading LLMs, CoP reveals unprecedented safety risks by finding novel jailbreak prompts and improving the best-known single-turn attack success rate by up to 19.0 times.
  arXiv  Detail & Related papers  (2025-06-01T02:18:41Z)
• AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integration [40.350632196772466]
  This paper introduces AutoRedTeamer, a novel framework for fully automated, end-to-end red teaming against large language models (LLMs)<n>AutoRedTeamer combines a multi-agent architecture with a memory-guided attack selection mechanism to enable continuous discovery and integration of new attack vectors.<n>We demonstrate AutoRedTeamer's effectiveness across diverse evaluation settings, achieving 20% higher attack success rates on HarmBench against Llama-3.1-70B.
  arXiv  Detail & Related papers  (2025-03-20T00:13:04Z)
• Intelligent Mobile AI-Generated Content Services via Interactive Prompt Engineering and Dynamic Service Provisioning [55.641299901038316]
  AI-generated content can organize collaborative Mobile AIGC Service Providers (MASPs) at network edges to provide ubiquitous and customized content for resource-constrained users.<n>Such a paradigm faces two significant challenges: 1) raw prompts often lead to poor generation quality due to users' lack of experience with specific AIGC models, and 2) static service provisioning fails to efficiently utilize computational and communication resources.<n>We develop an interactive prompt engineering mechanism that leverages a Large Language Model (LLM) to generate customized prompt corpora and employs Inverse Reinforcement Learning (IRL) for policy imitation.
  arXiv  Detail & Related papers  (2025-02-17T03:05:20Z)
• A Multi-AI Agent System for Autonomous Optimization of Agentic AI Solutions via Iterative Refinement and LLM-Driven Feedback Loops [3.729242965449096]
  This paper introduces a framework for autonomously optimizing Agentic AI solutions across industries.<n>The framework achieves optimal performance without human input by autonomously generating and testing hypotheses.<n>Case studies show significant improvements in output quality, relevance, and actionability.
  arXiv  Detail & Related papers  (2024-12-22T20:08:04Z)
• Agent-as-a-Judge: Evaluate Agents with Agents [61.33974108405561]
  We introduce the Agent-as-a-Judge framework, wherein agentic systems are used to evaluate agentic systems. This is an organic extension of the LLM-as-a-Judge framework, incorporating agentic features that enable intermediate feedback for the entire task-solving process. We present DevAI, a new benchmark of 55 realistic automated AI development tasks.
  arXiv  Detail & Related papers  (2024-10-14T17:57:02Z)
• ProAgent: From Robotic Process Automation to Agentic Process Automation [87.0555252338361]
  Large Language Models (LLMs) have emerged human-like intelligence. This paper introduces Agentic Process Automation (APA), a groundbreaking automation paradigm using LLM-based agents for advanced automation. We then instantiate ProAgent, an agent designed to craft from human instructions and make intricate decisions by coordinating specialized agents.
  arXiv  Detail & Related papers  (2023-11-02T14:32:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.