Do Agents Dream of Root Shells? Partial-Credit Evaluation of LLM Agents in Capture The Flag Challenges

• URL: http://arxiv.org/abs/2604.19354v1
• Date: Tue, 21 Apr 2026 11:35:33 GMT
• Title: Do Agents Dream of Root Shells? Partial-Credit Evaluation of LLM Agents in Capture The Flag Challenges
• Abstract summary: We present DeepRed, an open-source benchmark for evaluating Large Language Model (LLM) agents on realistic Capture The Flag (CTF) challenges.<n>DeepRed places an agent in a Kali attacker environment with terminal tools and optional web search, connected over a private network to a target challenge, and records full execution traces for analysis.<n>Using DeepRed, we benchmark ten commercially summarise LLMs on ten VM-based CTF challenges spanning different challenge categories.<n>The results indicate that the best model achieves only 35% average checkpoint completion, performing strongest on common challenge types and weakest on tasks requiring non-standard discovery
• Score: 12.212097872268766
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Model (LLM) agents are increasingly proposed for autonomous cybersecurity tasks, but their capabilities in realistic offensive settings remain poorly understood. We present DeepRed, an open-source benchmark for evaluating LLM-based agents on realistic Capture The Flag (CTF) challenges in isolated virtualized environments. DeepRed places an agent in a Kali attacker environment with terminal tools and optional web search, connected over a private network to a target challenge, and records full execution traces for analysis. To move beyond binary solved/unsolved outcomes, we introduce a partial-credit scoring method based on challenge-specific checkpoints derived from public writeups, together with an automated summarise-then-judge labelling pipeline for assigning checkpoint completion from logs. Using DeepRed, we benchmark ten commercially accessible LLMs on ten VM-based CTF challenges spanning different challenge categories. The results indicate that current agents remain limited: the best model achieves only 35% average checkpoint completion, performing strongest on common challenge types and weakest on tasks requiring non-standard discovery and longer-horizon adaptation.

Related papers

• Evaluating Context Segmentation in Locally Deployable SLMs for Cybersecurity CTF Tasks [0.0]
  We introduce context segmentation, a two-level agentic framework that divides complex exploitation tasks into manageable, contextually isolated sub-problems.<n>We demonstrate that for the E4B model, our strategy acts as an intelligent search, achieving competitive rewards with superior token efficiency compared to brute-force retries.
  arXiv  Detail & Related papers  (2026-09-11T13:34:48Z)
• Staying on the Attack Path: Structured State for Long-Horizon Automated Penetration Testing [17.809156797437208]
  Intentest is an intent-graph-guided automated penetration testing agent.<n>It externalizes long-horizon state from the LLM's context window onto a persistent fact-intent directed acyclic graph.<n>It achieves an overall success rate of 88.2% and a success rate of 75.0% on hard tasks.
  arXiv  Detail & Related papers  (2026-09-07T11:09:01Z)
• NetConfArena: An Executable Benchmark for LLM Agents in Closed-Loop Network Configuration [14.126948395559987]
  We present NetConfArena, an executable benchmark for evaluating large language model (LLM) agents in closed-loop network configuration.<n>The benchmark relies on an LLM-assisted, emulation-grounded pipeline, which converts human-oriented network materials into reusable parameterized task templates.<n>We evaluate representative LLM agents on 480 task instances instantiated from 96 protocol-focused task templates, and show that failures are not limited to command errors.
  arXiv  Detail & Related papers  (2026-08-24T12:26:37Z)
• StainFlow: Entity-Stain Tracking and Evidence Linking for Process Rewards in GUI Agents [67.03593791535786]
  Reinforcement Learning (RL) has become a promising approach for improving GUI Agents in long-horizon, digital environments.<n>To mitigate this issue, recent studies introduce Process Reward Models (PRMs)<n>PRMs provide finer-grained training feedback through global milestone verification or local step-level evaluation.<n>We propose StainFlow, an entity-stain-flow process reward model for GUI Agents.
  arXiv  Detail & Related papers  (2026-06-05T08:17:28Z)
• What to Test Next: Interpretable Coverage Gap Discovery in Driving VLMs [52.50210189669399]
  Driving vision-language models (VLMs) must accurately understand scenes across diverse conditions defined by Operational Design Domains (ODDs)<n>We propose SliceScorer, a deterministic scoring rule for missing-slice recommendation.<n>We show that SliceNav surfaces high-risk coverage gaps more effectively than prior slice-discovery methods.
  arXiv  Detail & Related papers  (2026-06-01T03:18:01Z)
• AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation [71.49152943451328]
  We introduce a benchmark AJ-Bench to evaluate Agent-as-a-Judge across three domains-search, data systems, and graphical user interfaces-comprising 155 tasks and 516 annotated trajectories.<n>Experiments demonstrate consistent performance gains over LLM-as-a-Judge baselines, while also revealing substantial open challenges in agent-based verification.
  arXiv  Detail & Related papers  (2026-04-20T13:23:38Z)
• UltraHorizon: Benchmarking Agent Capabilities in Ultra Long-Horizon Scenarios [63.67884284105684]
  We introduce textbfUltraHorizon, a novel benchmark that measures the foundational capabilities essential for complex real-world challenges.<n>Agents are designed in long-horizon discovery tasks where they must iteratively uncover hidden rules.<n>Our experiments reveal that LLM-agents consistently underperform in these settings, whereas human participants achieve higher scores.
  arXiv  Detail & Related papers  (2025-09-26T02:04:00Z)
• Autonomous Penetration Testing: Solving Capture-the-Flag Challenges with LLMs [0.0]
  This study evaluates the ability of GPT-4o to autonomously solve beginner-level offensive security tasks by connecting the model to OverTheWire's Bandit capture-the-flag game.<n>Of the 25 levels that were technically compatible with a single-command SSH framework, GPT-4o solved 18 unaided and another two after minimal prompt hints for an overall 80% success rate.
  arXiv  Detail & Related papers  (2025-08-01T20:11:58Z)
• ExCyTIn-Bench: Evaluating LLM agents on Cyber Threat Investigation [15.227180564453016]
  We present ExCyTIn-Bench, the first benchmark to evaluate an LLM agent x on the task of Cyber Threat Investigation.<n>We leverage security logs extracted with expert-crafted detection logic to build threat investigation graphs.<n>We generate questions with LLMs using paired nodes on the graph, taking the start node as background context and the end node as answer.
  arXiv  Detail & Related papers  (2025-07-14T17:06:26Z)
• LocationReasoner: Evaluating LLMs on Real-World Site Selection Reasoning [9.738040108018835]
  We introduce LocationReasoner, a benchmark designed to evaluate large language models' reasoning abilities in the context of real-world site selection.<n>The benchmark covers carefully crafted queries of varying difficulty levels and is supported by a sandbox environment with in-house tools for constraint-based location search.<n>Extensive evaluations on real-world site selection data from Boston, New York, and Tampa reveal that state-of-the-art reasoning models offer limited improvement over their non-reasoning predecessors.
  arXiv  Detail & Related papers  (2025-06-16T16:23:56Z)
• How Robust Are Router-LLMs? Analysis of the Fragility of LLM Routing Capabilities [62.474732677086855]
  Large language model (LLM) routing has emerged as a crucial strategy for balancing computational costs with performance.<n>We propose the DSC benchmark: Diverse, Simple, and Categorized, an evaluation framework that categorizes router performance across a broad spectrum of query types.
  arXiv  Detail & Related papers  (2025-03-20T19:52:30Z)
• Can LLMs Hack Enterprise Networks? Autonomous Assumed Breach Penetration-Testing Active Directory Networks [1.3124479769761592]
  We introduce a novel prototype designed to employ Large Language Model (LLM)-driven autonomous systems.<n>Our system represents the first demonstration of a fully autonomous, LLM-driven framework capable of compromising accounts.<n>We find that the associated costs are competitive with, and often significantly lower than, those incurred by professional human pen-testers.
  arXiv  Detail & Related papers  (2025-02-06T17:12:43Z)
• AutoPT: How Far Are We from the End2End Automated Web Penetration Testing? [54.65079443902714]
  We introduce AutoPT, an automated penetration testing agent based on the principle of PSM driven by LLMs. Our results show that AutoPT outperforms the baseline framework ReAct on the GPT-4o mini model.
  arXiv  Detail & Related papers  (2024-11-02T13:24:30Z)
• MMAU: A Holistic Benchmark of Agent Capabilities Across Diverse Domains [54.117238759317004]
  Massive Multitask Agent Understanding (MMAU) benchmark features comprehensive offline tasks that eliminate the need for complex environment setups. It evaluates models across five domains, including Tool-use, Directed Acyclic Graph (DAG) QA, Data Science and Machine Learning coding, Contest-level programming and Mathematics. With a total of 20 meticulously designed tasks encompassing over 3K distinct prompts, MMAU provides a comprehensive framework for evaluating the strengths and limitations of LLM agents.
  arXiv  Detail & Related papers  (2024-07-18T00:58:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.