CTFusion: A CTF-based Benchmark for LLM Agent Evaluation

• URL: http://arxiv.org/abs/2605.11504v1
• Date: Tue, 12 May 2026 04:23:42 GMT
• Title: CTFusion: A CTF-based Benchmark for LLM Agent Evaluation
• Abstract summary: We present CTFusion, a streaming evaluation framework built on Live CTFs.<n>We demonstrate that existing CTF benchmarks can be unreliable in assessing LLM-based agents.<n>We release CTFusion as open source to foster future research in this area.
• Score: 15.091299960425857
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in Large Language Models (LLMs) have enabled agentic systems for complex, multi-step tasks; cybersecurity is emerging as a prominent application. To evaluate such agents, researchers widely adopt Capture The Flag (CTF) benchmarks. However, current CTF benchmarks reuse existing challenges, which exposes them to data contamination and potential cheating. Notably, we confirmed these issues in practice by integrating web search tools into an existing agent. To address these limitations, we present CTFusion, a streaming evaluation framework built on Live CTFs. To achieve this, CTFusion preserves per-agent independence under a single team account and reduces competition impact by forwarding only the first correct flag per challenge. Moreover, we implement CTFusion as a Model Context Protocol (MCP) server on the widely used CTFd platform, which offers broad applicability to diverse CTF events and agent types. Through experiments with three LLMs, two agents, and five Live CTFs, we demonstrate that existing CTF benchmarks can be unreliable in assessing LLM-based agents, while CTFusion can serve as a robust solution for evaluating cybersecurity agents. We release CTFusion as open source to foster future research in this area.

Related papers

• AgentCompass: A Unified Evaluation Infrastructure for Agent Capabilities [55.23913617185508]
  Agent is an open-source, lightweight, and infrastructure for evaluating Large Language Models (LLMs)-based agents.<n>It organizes the evaluation process around three independent components, namely Benchmark, Harness and Environment.<n>It features a fault-tolerant runtime and comprehensive trajectory analysis tools to transparently diagnose failure modes like reward-hacking.
  arXiv  Detail & Related papers  (2026-07-15T11:11:17Z)
• The Meta-Agent Challenge: Are Current Agents Capable of Autonomous Agent Development? [80.24951682268332]
  We introduce the Meta-Agent Challenge (MAC), an evaluation framework designed to test the capacity of frontier models for autonomous agent development.<n>To ensure evaluation integrity, this framework is secured by multi-layer defenses against reward hacking.<n>We demonstrate that meta-agents rarely match human-engineered baseline policies, and the few that do are dominated by proprietary frontier models.
  arXiv  Detail & Related papers  (2026-06-03T04:58:17Z)
• FAMA: Failure-Aware Meta-Agentic Framework for Open-Source LLMs in Interactive Tool Use Environments [60.3427704389541]
  Large Language Models are being increasingly deployed as the decision-making core of autonomous agents.<n>Yet, in conversational benchmarks, these agents frequently fail due to the cascading effects of incorrect decision-making.<n>We present the Failure-Aware Meta-Agentic framework to tackle these challenges.
  arXiv  Detail & Related papers  (2026-04-28T02:21:53Z)
• STRIATUM-CTF: A Protocol-Driven Agentic Framework for General-Purpose CTF Solving [1.266832699428407]
  We introduce STRIATUM-CTF (A Search-based Test-time Reasoning Inference Agent for Tactical Utility Maximization in Cybersecurity), a modular agentic framework built upon the Model Context Protocol (MCP)<n>By standardizing tool interfaces for system introspection, decompilation, and runtime debug, STRIATUM-CTF enables the agent to maintain a coherent context window across extended exploit trajectories.<n>Our system participated in a university-hosted Capture-the-Flag (CTF) competition in late 2025, where it operated autonomously to identify and exploit vulnerabilities in real-time.
  arXiv  Detail & Related papers  (2026-03-23T21:17:26Z)
• Breaking User-Centric Agency: A Tri-Party Framework for Agent-Based Recommendation [59.362016745945375]
  We propose the first Tri-party LLM-agent Recommendation framework (TriRec)<n>TriRec explicitly coordinates user utility, item exposure, and platform-level fairness.<n>We show consistent gains in accuracy, fairness, and item-level utility.
  arXiv  Detail & Related papers  (2026-03-11T11:40:13Z)
• Capture the Flags: Family-Based Evaluation of Agentic LLMs via Semantics-Preserving Transformations [9.234598988803407]
  Agentic large language models (LLMs) are increasingly evaluated on cybersecurity tasks using capture-the-flag benchmarks.<n>We introduce CTF challenge families, whereby a single CTF is used as the basis for generating a family of semantically-equivalent challenges.<n>We introduce a new tool, Evolve-CTF, that generates CTF families from Python challenges using a range of transformations.
  arXiv  Detail & Related papers  (2026-02-05T10:30:57Z)
• Towards Effective Offensive Security LLM Agents: Hyperparameter Tuning, LLM as a Judge, and a Lightweight CTF Benchmark [18.653746750609546]
  We systematically investigate the key factors that drive agent success and provide a detailed recipe for building effective LLM-based offensive security agents.<n>We present CTFJudge, a framework leveraging LLM as a judge to analyze agent trajectories and provide granular evaluation across CTF solving steps.<n>For rapid evaluation, we present CTFTiny, a curated benchmark of 50 representative CTF challenges across binary exploitation, web, reverse engineering, forensics, and cryptography.
  arXiv  Detail & Related papers  (2025-08-05T03:25:09Z)
• Why Do Multi-Agent LLM Systems Fail? [87.90075668488434]
  We introduce MAST-Data, a comprehensive dataset of 1600+ annotated traces collected across 7 popular MAS frameworks.<n>We build the first Multi-Agent System Failure taxonomy (MAST)<n>We leverage MAST and MAST-Data to analyze failure patterns across models (GPT4, Claude 3, Qwen2.5, CodeLlama) and tasks (coding, math, general agent)
  arXiv  Detail & Related papers  (2025-03-17T19:04:38Z)
• D-CIPHER: Dynamic Collaborative Intelligent Multi-Agent System with Planner and Heterogeneous Executors for Offensive Security [22.86304661035188]
  D-CIPHER is a multi-agent framework for collaborative cybersecurity CTF solving.<n>It integrates agents with distinct roles with dynamic feedback loops to enhance reasoning on complex tasks.<n>It achieves state-of-the-art performance on three benchmarks: 22.0% on NYU CTF Bench, 22.5% on Cybench, and 44.0% on HackTheBox.
  arXiv  Detail & Related papers  (2025-02-15T23:43:18Z)
• EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilities [46.34031902647788]
  We present EnIGMA, an LM agent for autonomously solving Capture The Flag (CTF) challenges.<n>We introduce new tools and interfaces to improve the agent's ability to find and exploit security vulnerabilities.<n> Empirical analysis on 390 CTF challenges demonstrate that these new tools and interfaces substantially improve our agent's performance.
  arXiv  Detail & Related papers  (2024-09-24T15:06:01Z)
• Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for Large Language Models [56.00992369295851]
  Open-sourced Large Language Models (LLMs) have achieved great success in various NLP tasks, however, they are still far inferior to API-based models when acting as agents. This paper delivers three key observations: (1) the current agent training corpus is entangled with both formats following and agent reasoning, which significantly shifts from the distribution of its pre-training data; (2) LLMs exhibit different learning speeds on the capabilities required by agent tasks; and (3) current approaches have side-effects when improving agent abilities by introducing hallucinations. We propose Agent-FLAN to effectively Fine-tune LANguage models for Agents.
  arXiv  Detail & Related papers  (2024-03-19T16:26:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.