D-CIPHER: Dynamic Collaborative Intelligent Multi-Agent System with Planner and Heterogeneous Executors for Offensive Security

• URL: http://arxiv.org/abs/2502.10931v2
• Date: Sun, 11 May 2025 03:59:37 GMT
• Title: D-CIPHER: Dynamic Collaborative Intelligent Multi-Agent System with Planner and Heterogeneous Executors for Offensive Security
• Abstract summary: D-CIPHER is a multi-agent framework for collaborative cybersecurity CTF solving.<n>It integrates agents with distinct roles with dynamic feedback loops to enhance reasoning on complex tasks.<n>It achieves state-of-the-art performance on three benchmarks: 22.0% on NYU CTF Bench, 22.5% on Cybench, and 44.0% on HackTheBox.
• Score: 22.86304661035188
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) have been used in cybersecurity such as autonomous security analysis or penetration testing. Capture the Flag (CTF) challenges serve as benchmarks to assess automated task-planning abilities of LLM agents for cybersecurity. Early attempts to apply LLMs for solving CTF challenges used single-agent systems, where feedback was restricted to a single reasoning-action loop. This approach was inadequate for complex CTF tasks. Inspired by real-world CTF competitions, where teams of experts collaborate, we introduce the D-CIPHER LLM multi-agent framework for collaborative CTF solving. D-CIPHER integrates agents with distinct roles with dynamic feedback loops to enhance reasoning on complex tasks. It introduces the Planner-Executor agent system, consisting of a Planner agent for overall problem-solving along with multiple heterogeneous Executor agents for individual tasks, facilitating efficient allocation of responsibilities among the agents. Additionally, D-CIPHER incorporates an Auto-prompter agent to improve problem-solving by auto-generating a highly relevant initial prompt. We evaluate D-CIPHER on multiple CTF benchmarks and LLM models via comprehensive studies to highlight the impact of our enhancements. Additionally, we manually map the CTFs in NYU CTF Bench to MITRE ATT&CK techniques that apply for a comprehensive evaluation of D-CIPHER's offensive security capability. D-CIPHER achieves state-of-the-art performance on three benchmarks: 22.0% on NYU CTF Bench, 22.5% on Cybench, and 44.0% on HackTheBox, which is 2.5% to 8.5% better than previous work. D-CIPHER solves 65% more ATT&CK techniques compared to previous work, demonstrating stronger offensive capability.

Related papers

• CTFusion: A CTF-based Benchmark for LLM Agent Evaluation [15.091299960425857]
  We present CTFusion, a streaming evaluation framework built on Live CTFs.<n>We demonstrate that existing CTF benchmarks can be unreliable in assessing LLM-based agents.<n>We release CTFusion as open source to foster future research in this area.
  arXiv  Detail & Related papers  (2026-05-12T04:23:42Z)
• Autonomous LLM Agents & CTFs: A Second Look [2.475524451473685]
  Large Language Model (LLM) agents are increasingly proposed to automate offensive security tasks.<n>We engineer different agent architectures of increasing complexity and modularity on 30 web-based Capture-the-Flag (CTF) challenges.<n>We instantiate these agents with multiple LLM backbones, and compare them with claude-code, a general-purpose agent that automatically determines its internal architecture.
  arXiv  Detail & Related papers  (2026-04-29T09:42:36Z)
• Agent Q-Mix: Selecting the Right Action for LLM Multi-Agent Systems through Reinforcement Learning [68.85408801740228]
  We propose textbfAgent Q-Mix, a reinforcement learning framework that reformulates topology selection as a cooperative Multi-Agent Reinforcement Learning problem.<n>Our method learns decentralized communication decisions using Qmix value factorization, where each agent selects from a set of communication actions that jointly induce a round-wise communication graph.<n>Agent Q-Mix achieves the highest average accuracy compared to existing methods while demonstrating superior token efficiency and robustness against agent failure.
  arXiv  Detail & Related papers  (2026-04-01T00:38:24Z)
• STRIATUM-CTF: A Protocol-Driven Agentic Framework for General-Purpose CTF Solving [1.266832699428407]
  We introduce STRIATUM-CTF (A Search-based Test-time Reasoning Inference Agent for Tactical Utility Maximization in Cybersecurity), a modular agentic framework built upon the Model Context Protocol (MCP)<n>By standardizing tool interfaces for system introspection, decompilation, and runtime debug, STRIATUM-CTF enables the agent to maintain a coherent context window across extended exploit trajectories.<n>Our system participated in a university-hosted Capture-the-Flag (CTF) competition in late 2025, where it operated autonomously to identify and exploit vulnerabilities in real-time.
  arXiv  Detail & Related papers  (2026-03-23T21:17:26Z)
• KARL: Knowledge Agents via Reinforcement Learning [63.627906947205624]
  We present a system for training enterprise search agents via reinforcement learning.<n> KARLBench is a multi-capability evaluation suite spanning six distinct search regimes.<n>We show that models trained across heterogeneous search behaviors generalize substantially better than those optimized for any single benchmark.
  arXiv  Detail & Related papers  (2026-03-05T14:30:25Z)
• AgentIF-OneDay: A Task-level Instruction-Following Benchmark for General AI Agents in Daily Scenarios [49.90735676070039]
  The capacity of AI agents to effectively handle tasks of increasing duration and complexity continues to grow.<n>We argue that current evaluations prioritize increasing task difficulty without sufficiently addressing the diversity of agentic tasks.<n>We propose AgentIF-OneDay, aimed at determining whether general users can utilize natural language instructions and AI agents to complete a diverse array of daily tasks.
  arXiv  Detail & Related papers  (2026-01-28T13:49:18Z)
• Collaborative Multi-Agent Test-Time Reinforcement Learning for Reasoning [112.16686518063456]
  We introduce textbfMulti-Agent Test-Time Reinforcement Learning (MATTRL), a framework that injects structured textual experience into multi-agent deliberation at inference time.<n>MATTRL forms a multi-expert team of specialists for multi-turn discussions, retrieves and integrates test-time experiences, and reaches consensus for final decision-making.<n>Across challenging benchmarks in medicine, math, and education, MATTRL improves accuracy by an average of 3.67% over a multi-agent baseline, and by 8.67% over comparable single-agent baselines
  arXiv  Detail & Related papers  (2026-01-14T17:57:43Z)
• Scaling Agents via Continual Pre-training [80.97989245493326]
  We propose incorporating Agentic Continual Pre-training (Agentic CPT) into the deep research agents training pipeline to build powerful agentic foundational models.<n>We evaluate our AgentFounder-30B on 10 benchmarks and achieve state-of-the-art performance while retains strong tool-use ability.
  arXiv  Detail & Related papers  (2025-09-16T17:57:19Z)
• Towards Effective Offensive Security LLM Agents: Hyperparameter Tuning, LLM as a Judge, and a Lightweight CTF Benchmark [18.653746750609546]
  We systematically investigate the key factors that drive agent success and provide a detailed recipe for building effective LLM-based offensive security agents.<n>We present CTFJudge, a framework leveraging LLM as a judge to analyze agent trajectories and provide granular evaluation across CTF solving steps.<n>For rapid evaluation, we present CTFTiny, a curated benchmark of 50 representative CTF challenges across binary exploitation, web, reverse engineering, forensics, and cryptography.
  arXiv  Detail & Related papers  (2025-08-05T03:25:09Z)
• Measuring and Augmenting Large Language Models for Solving Capture-the-Flag Challenges [10.476975554297095]
  CTF competitions are crucial for cybersecurity education and training.<n>As large language models (LLMs) evolve, there is increasing interest in their ability to automate CTF challenge solving.<n>We propose CTFAgent, a novel framework for advancing CTF problem-solving.
  arXiv  Detail & Related papers  (2025-06-21T08:56:20Z)
• CRAKEN: Cybersecurity LLM Agent with Knowledge-Based Execution [22.86304661035188]
  Large Language Model (LLM) agents can automate cybersecurity tasks and can adapt to the evolving cybersecurity landscape without re-engineering.<n>But they have two key limitations: accessing latest cybersecurity expertise beyond training data, and integrating new knowledge into complex task planning.<n>We present CRAKEN, a knowledge-based LLM agent framework that improves cybersecurity capability through three core mechanisms.
  arXiv  Detail & Related papers  (2025-05-21T11:01:11Z)
• AegisLLM: Scaling Agentic Systems for Self-Reflective Defense in LLM Security [74.22452069013289]
  AegisLLM is a cooperative multi-agent defense against adversarial attacks and information leakage.<n>We show that scaling agentic reasoning system at test-time substantially enhances robustness without compromising model utility.<n> Comprehensive evaluations across key threat scenarios, including unlearning and jailbreaking, demonstrate the effectiveness of AegisLLM.
  arXiv  Detail & Related papers  (2025-04-29T17:36:05Z)
• Why Do Multi-Agent LLM Systems Fail? [91.39266556855513]
  We present MAST (Multi-Agent System Failure taxonomy), the first empirically grounded taxonomy designed to understand MAS failures. We analyze seven popular MAS frameworks across over 200 tasks, involving six expert human annotators. We identify 14 unique failure modes, organized into 3 overarching categories, (i) specification issues, (ii) inter-agent misalignment, and (iii) task verification.
  arXiv  Detail & Related papers  (2025-03-17T19:04:38Z)
• Privacy-Enhancing Paradigms within Federated Multi-Agent Systems [47.76990892943637]
  LLM-based Multi-Agent Systems (MAS) have proven highly effective in solving complex problems by integrating multiple agents, each performing different roles. In this paper, we introduce the concept of Federated MAS, highlighting the fundamental differences between Federated MAS and traditional FL. We then identify key challenges in developing Federated MAS, including: 1) heterogeneous privacy protocols among agents, 2) structural differences in multi-party conversations, and 3) dynamic conversational network structures. To address these challenges, we propose Embedded Privacy-Enhancing Agents (EPEAgent), an innovative solution that integrates seamlessly into the Retrieval-Augmented Generation phase and the
  arXiv  Detail & Related papers  (2025-03-11T08:38:45Z)
• Agent-Oriented Planning in Multi-Agent Systems [54.429028104022066]
  We propose AOP, a novel framework for agent-oriented planning in multi-agent systems. In this study, we identify three critical design principles of agent-oriented planning, including solvability, completeness, and non-redundancy. Extensive experiments demonstrate the advancement of AOP in solving real-world problems compared to both single-agent systems and existing planning strategies for multi-agent systems.
  arXiv  Detail & Related papers  (2024-10-03T04:07:51Z)
• Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilities [46.34031902647788]
  We present EnIGMA, an LM agent for autonomously solving Capture The Flag (CTF) challenges.<n>We introduce new tools and interfaces to improve the agent's ability to find and exploit security vulnerabilities.<n> Empirical analysis on 390 CTF challenges demonstrate that these new tools and interfaces substantially improve our agent's performance.
  arXiv  Detail & Related papers  (2024-09-24T15:06:01Z)
• Textualized Agent-Style Reasoning for Complex Tasks by Multiple Round LLM Generation [49.27250832754313]
  We present AgentCOT, a llm-based autonomous agent framework. At each step, AgentCOT selects an action and executes it to yield an intermediate result with supporting evidence. We introduce two new strategies to enhance the performance of AgentCOT.
  arXiv  Detail & Related papers  (2024-09-19T02:20:06Z)
• Collaboration Dynamics and Reliability Challenges of Multi-Agent LLM Systems in Finite Element Analysis [3.437656066916039]
  How interagent dynamics influence reasoning quality and verification reliability remains unclear.<n>We study these mechanisms using an AutoGen-based multi-agent framework for linear-elastic Finite Element Analysis (FEA)<n>From 1,120 controlled trials, we find that collaboration effectiveness depends more on functional complementarity than team size.
  arXiv  Detail & Related papers  (2024-08-23T23:11:08Z)
• Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents [44.34340798542]
  Large Language Models (LLMs) have shown remarkable capabilities in natural language tasks requiring complex reasoning. Traditional supervised pre-training on static datasets falls short in enabling autonomous agent capabilities. We propose a framework that combines guided Monte Carlo Tree Search (MCTS) search with a self-critique mechanism and iterative fine-tuning on agent interactions.
  arXiv  Detail & Related papers  (2024-08-13T20:52:13Z)
• On the Resilience of LLM-Based Multi-Agent Collaboration with Faulty Agents [58.79302663733703]
  Large language model-based multi-agent systems have shown great abilities across various tasks due to the collaboration of expert agents.<n>However, the impact of clumsy or even malicious agents, on the overall performance of the system remains underexplored.<n>This paper investigates what is the resilience of various system structures under faulty agents.
  arXiv  Detail & Related papers  (2024-08-02T03:25:20Z)
• EvoAgent: Towards Automatic Multi-Agent Generation via Evolutionary Algorithms [55.77492625524141]
  EvoAgent is a generic method to automatically extend specialized agents to multi-agent systems. We show that EvoAgent can significantly enhance the task-solving capability of LLM-based agents.
  arXiv  Detail & Related papers  (2024-06-20T11:49:23Z)
• Adaptive In-conversation Team Building for Language Model Agents [33.03550687362213]
  Leveraging multiple large language model (LLM) agents has shown to be a promising approach for tackling complex tasks. Our new adaptive team-building paradigm offers a flexible solution, realized through a novel agent design named Captain Agent. A comprehensive evaluation across six real-world scenarios demonstrates that Captain Agent significantly outperforms existing multi-agent methods.
  arXiv  Detail & Related papers  (2024-05-29T18:08:37Z)
• Smurfs: Leveraging Multiple Proficiency Agents with Context-Efficiency for Tool Planning [14.635361844362794]
  Smurfs' is a cutting-edge multi-agent framework designed to revolutionize the application of large language models. Smurfs can enhance the model's ability to solve complex tasks at no additional cost.
  arXiv  Detail & Related papers  (2024-05-09T17:49:04Z)
• Learning to Use Tools via Cooperative and Interactive Agents [58.77710337157665]
  Tool learning empowers large language models (LLMs) as agents to use external tools and extend their utility. We propose ConAgents, a Cooperative and interactive Agents framework, which coordinates three specialized agents for tool selection, tool execution, and action calibration separately. Our experiments on three datasets show that the LLMs, when equipped with ConAgents, outperform baselines with substantial improvement.
  arXiv  Detail & Related papers  (2024-03-05T15:08:16Z)
• A Dynamic LLM-Powered Agent Network for Task-Oriented Agent Collaboration [55.35849138235116]
  We propose automatically selecting a team of agents from candidates to collaborate in a dynamic communication structure toward different tasks and domains. Specifically, we build a framework named Dynamic LLM-Powered Agent Network ($textDyLAN$) for LLM-powered agent collaboration. We demonstrate that DyLAN outperforms strong baselines in code generation, decision-making, general reasoning, and arithmetic reasoning tasks with moderate computational cost.
  arXiv  Detail & Related papers  (2023-10-03T16:05:48Z)
• Robustness Testing for Multi-Agent Reinforcement Learning: State Perturbations on Critical Agents [2.5204420653245245]
  Multi-Agent Reinforcement Learning (MARL) has been widely applied in many fields such as smart traffic and unmanned aerial vehicles. This work proposes a novel Robustness Testing framework for MARL that attacks states of Critical Agents.
  arXiv  Detail & Related papers  (2023-06-09T02:26:28Z)
• Learning to Solve Complex Tasks by Talking to Agents [39.08818632689814]
  Humans often solve complex problems by interacting with existing agents, such as AI assistants, that can solve simpler sub-tasks. Common NLP benchmarks aim for the development of self-sufficient models for every task. We propose a new benchmark called CommaQA that contains three kinds of complex reasoning tasks designed to be solved by talking'' to four agents with different capabilities.
  arXiv  Detail & Related papers  (2021-10-16T10:37:34Z)
• FACMAC: Factored Multi-Agent Centralised Policy Gradients [103.30380537282517]
  We propose FACtored Multi-Agent Centralised policy gradients (FACMAC) It is a new method for cooperative multi-agent reinforcement learning in both discrete and continuous action spaces. We evaluate FACMAC on variants of the multi-agent particle environments, a novel multi-agent MuJoCo benchmark, and a challenging set of StarCraft II micromanagement tasks.
  arXiv  Detail & Related papers  (2020-03-14T21:29:09Z)
• Counterfactual Multi-Agent Policy Gradients [47.45255170608965]
  We propose a new multi-agent actor-critic method called counterfactual multi-agent (COMA) policy gradients.<n>COMA uses a centralised critic to estimate the Q-function and decentralised actors to optimise the agents' policies.<n>We evaluate COMA in the testbed of StarCraft unit micromanagement, using a decentralised variant with significant partial observability.
  arXiv  Detail & Related papers  (2017-05-24T18:52:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.