Autonomous LLM Agents & CTFs: A Second Look

• URL: http://arxiv.org/abs/2605.21497v1
• Date: Wed, 29 Apr 2026 09:42:36 GMT
• Title: Autonomous LLM Agents & CTFs: A Second Look
• Abstract summary: Large Language Model (LLM) agents are increasingly proposed to automate offensive security tasks.<n>We engineer different agent architectures of increasing complexity and modularity on 30 web-based Capture-the-Flag (CTF) challenges.<n>We instantiate these agents with multiple LLM backbones, and compare them with claude-code, a general-purpose agent that automatically determines its internal architecture.
• Score: 2.475524451473685
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Model (LLM) agents are increasingly proposed to automate offensive security tasks, with recent studies reporting near human-level success rates in Capture-the-Flag (CTF) challenges. We here revisit these results, providing a second look at these claims. We engineer different agent architectures of increasing complexity and modularity on 30 web-based CTFs challenges spanning 14 vulnerability classes. We instantiate these agents with multiple LLM backbones, and compare them with claude-code, a general-purpose agent that automatically determines its internal architecture. Our evaluation yields three main findings. First, claude-code achieves performance comparable to the engineered architectures (19/30 solved tasks), suggesting that general-purpose agents are strong baselines for offensive security tasks. Second, both our architectures and claude-code struggle in the same challenge categories, revealing persistent barriers that keep current agents below human-level capability. Third, by leveraging our manually designed architectures we can systematically measure the impact of additional components, finding that structured orchestration of specialized roles outperforms monolithic designs, improving run-to-run consistency, and reducing execution costs.

Related papers

• A Survey of LLM-Driven Penetration Testing: Taxonomy, Co-Evolution, and Open Challenges [39.189948044905364]
  We organize the literature into six categories: evaluation benchmarks, general-purpose systems, domain-specific frameworks, CTF-based systems, defense-oriented research, and surveys.<n>First, RLVR marks a shift in capability acquisition from imitation of expert demonstrations to reward-driven self-improvement.<n>Second, CTF platforms have evolved from evaluation testbeds into dual-purpose infrastructure for both agent evaluation and RL training.
  arXiv  Detail & Related papers  (2026-07-01T13:24:25Z)
• The Illusion of Multi-Agent Advantage [63.92466143571383]
  We show that expert-architected Multi-Agent Systems (MAS) consistently outperforms automatically generated architectures in raw performance and cost-efficiency.<n>Existing evaluation frameworks mask critical architectural gaps and inefficiencies by failing to account for the marginal utility of increased computational cost.
  arXiv  Detail & Related papers  (2026-06-11T07:39:24Z)
• Architecture Matters for Multi-Agent Security [1.699014226195857]
  We study how multi-agent systems shape the tradeoff between task performance and attack resistance.<n>We find that multi-agent architectures are more vulnerable than standalone agents in the majority of configurations.
  arXiv  Detail & Related papers  (2026-04-25T22:25:58Z)
• Architecture-Aware Multi-Design Generation for Repository-Level Feature Addition [53.50448142467294]
  RAIM is a multi-design and architecture-aware framework for repository-level feature addition.<n>It shifts away from linear patching by generating multiple diverse implementation designs.<n>Experiments on the NoCode-bench Verified dataset demonstrate that RAIM establishes a new state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-03-02T12:50:40Z)
• MagicAgent: Towards Generalized Agent Planning [73.21129030631421]
  We present textbfMagicAgent, a series of foundation models specifically designed for generalized agent planning.<n>We introduce a lightweight and scalable synthetic data framework that generates high-quality trajectories across diverse planning tasks.<n>We show that MagicAgent-32B and MagicAgent-30B-A3B achieve superior performance across diverse open-source benchmarks.
  arXiv  Detail & Related papers  (2026-02-22T01:39:16Z)
• Yunque DeepResearch Technical Report [12.184074646161223]
  Yunque DeepResearch is a hierarchical, modular, and robust framework for deep research.<n>It achieves state-of-the-art performance across a range of agentic deep research benchmarks.<n>We open-source the framework, reproducible implementations, and application cases to empower the community.
  arXiv  Detail & Related papers  (2026-01-27T13:10:00Z)
• An Empirical Study on Failures in Automated Issue Solving [12.571536148821144]
  We analyze the performance and efficiency of three SOTA tools, spanning both pipeline-based and agentic architectures, in automated issue solving tasks of SWE-Bench-Verified.<n>To move from high-level performance metrics to underlying cause analysis, we conducted a systematic manual analysis of 150 failed instances.<n>The results reveal distinct failure fingerprints between the two architectural paradigms, with the majority of agentic failures stemming from flawed reasoning and cognitive deadlocks.
  arXiv  Detail & Related papers  (2025-09-17T13:07:52Z)
• AgentSwift: Efficient LLM Agent Design via Value-guided Hierarchical Search [58.98450205734779]
  Large language model (LLM) agents have demonstrated strong capabilities across diverse domains.<n>Existing agent search methods suffer from three major limitations.<n>We introduce a comprehensive framework to address these challenges.
  arXiv  Detail & Related papers  (2025-06-06T12:07:23Z)
• OWL: Optimized Workforce Learning for General Multi-Agent Assistance in Real-World Task Automation [65.15955645757705]
  We introduce Workforce, a hierarchical multi-agent framework that decouples strategic planning from specialized execution.<n>During inference, Workforce seamlessly adapts to new domains by adding or modifying worker agents.<n>For training, we introduce optimized Workforce Learning (OWL), which improves generalization across domains.
  arXiv  Detail & Related papers  (2025-05-29T17:51:58Z)
• Topological Structure Learning Should Be A Research Priority for LLM-Based Multi-Agent Systems [69.95482609893236]
  Large Language Model-based Multi-Agent Systems (MASs) have emerged as a powerful paradigm for tackling complex tasks through collaborative intelligence.<n>We call for a paradigm shift toward emphtopology-aware MASs that explicitly model and dynamically optimize the structure of inter-agent interactions.
  arXiv  Detail & Related papers  (2025-05-28T15:20:09Z)
• Why Do Multi-Agent LLM Systems Fail? [87.90075668488434]
  We introduce MAST-Data, a comprehensive dataset of 1600+ annotated traces collected across 7 popular MAS frameworks.<n>We build the first Multi-Agent System Failure taxonomy (MAST)<n>We leverage MAST and MAST-Data to analyze failure patterns across models (GPT4, Claude 3, Qwen2.5, CodeLlama) and tasks (coding, math, general agent)
  arXiv  Detail & Related papers  (2025-03-17T19:04:38Z)
• D-CIPHER: Dynamic Collaborative Intelligent Multi-Agent System with Planner and Heterogeneous Executors for Offensive Security [22.86304661035188]
  D-CIPHER is a multi-agent framework for collaborative cybersecurity CTF solving.<n>It integrates agents with distinct roles with dynamic feedback loops to enhance reasoning on complex tasks.<n>It achieves state-of-the-art performance on three benchmarks: 22.0% on NYU CTF Bench, 22.5% on Cybench, and 44.0% on HackTheBox.
  arXiv  Detail & Related papers  (2025-02-15T23:43:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.