ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?

• URL: http://arxiv.org/abs/2605.11086v1
• Date: Mon, 11 May 2026 18:00:14 GMT
• Title: ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?
• Abstract summary: Exploitation is a challenging task because it requires low-level program reasoning.<n>Despite its importance and diagnostic value, exploitation remains under-evaluated.<n>We introduce ExploitGym, a large-scale, diverse, realistic benchmark on the exploitation capabilities of AI agents.
• Score: 92.21756459993695
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: AI agents are rapidly gaining capabilities that could significantly reshape cybersecurity, making rigorous evaluation urgent. A critical capability is exploitation: turning a vulnerability, which is not yet an attack, into a concrete security impact, such as unauthorized file access or code execution. Exploitation is a particularly challenging task because it requires low-level program reasoning (e.g., about memory layout), runtime adaptation, and sustained progress over long horizons. Meanwhile, it is inherently dual-use, supporting defensive workflows while lowering the barrier for offense. Despite its importance and diagnostic value, exploitation remains under-evaluated. To address this gap, we introduce ExploitGym, a large-scale, diverse, realistic benchmark on the exploitation capabilities of AI agents. Given a program input that triggers a vulnerability, ExploitGym tasks agents with progressively extending it into a working exploit. The benchmark comprises 898 instances sourced from real-world vulnerabilities across three domains, including userspace programs, Google's V8 JavaScript engine, and the Linux kernel. We vary the security protections applied to each instance, isolating their impact on agent performance. All configurations are packaged in reproducible containerized environments. Our evaluation shows that while exploitation remains challenging, frontier models can successfully exploit a non-trivial fraction of vulnerabilities. For example, the strongest configurations are Anthropic's latest model Claude Mythos Preview and OpenAI's GPT-5.5, which produce working exploits for 157 and 120 instances, respectively. Notably, even with widely used defenses enabled, models retain non-trivial success rates. These results establish ExploitGym as an effective testbed for exploitation and highlight the growing cybersecurity risks posed by increasingly capable AI agents.

Related papers

• AgentCyberRange: Benchmarking Frontier AI Systems in Realistic Cyber Ranges [20.164879773235594]
  We introduce AgentCyberRange, the first open, multi-range infrastructure for measuring autonomous cyber attack capability in realistic cyber ranges.<n>It combines 110 vulnerabilities across 15 real web applications and 8 enterprise-like cyber ranges with 156 internal hosts, plus Cage, a toolchain for execution, orchestration, result collection, and verification.<n>We evaluate six frontier AI systems under matched prompts and budgets. GPT-5.5 with Codex performs best, solving 16.1% of web exploitation tasks and 31.7% of post-exploitation tasks.
  arXiv  Detail & Related papers  (2026-06-12T09:29:58Z)
• CyberGym-E2E: Scalable Real-World Benchmark for AI Agents' End-to-End Cybersecurity Capabilities [66.48570184296857]
  CyberGym-E2E is a large-scale and realistic end-to-end cybersecurity benchmark.<n>It comprehensively evaluates AI agents' abilities across the full lifecycle of vulnerability discovery, PoC generation, and patch generation.<n>The benchmark consists of 920 real-world vulnerabilities across 139 different open-source projects.
  arXiv  Detail & Related papers  (2026-06-03T05:06:37Z)
• A Multi-Agent Framework for Automated Exploit Generation with Constraint-Guided Comprehension and Reflection [30.539826266048014]
  Vulnsage is a multi-agent framework for automated Exploit Generation.<n>It generates 34.64% more exploits than state-of-the-art tools such as explodejs.<n>Vulnsage has successfully discovered and verified 146 zero-day vulnerabilities in real-world scenarios.
  arXiv  Detail & Related papers  (2026-04-06T19:47:17Z)
• Building Browser Agents: Architecture, Security, and Practical Solutions [0.0]
  This paper presents findings from building and operating a production browser agent.<n>Model capability does not limit agent performance; architectural decisions determine success or failure.<n>Security analysis of real-world incidents reveals prompt injection attacks make general-purpose autonomous operation fundamentally unsafe.
  arXiv  Detail & Related papers  (2025-11-22T12:18:35Z)
• HackWorld: Evaluating Computer-Use Agents on Exploiting Web Application Vulnerabilities [20.201614123811872]
  HackWorld is the first framework for evaluating computer-use agents' capabilities to exploit web application vulnerabilities via visual interaction.<n>It includes 36 real-world applications across 11 frameworks and 7 languages, featuring realistic flaws such as injection vulnerabilities, authentication bypasses, and unsafe input handling.<n>It tests CUAs' capacity to identify and exploit these weaknesses while navigating complex web interfaces.
  arXiv  Detail & Related papers  (2025-10-14T06:52:15Z)
• Cuckoo Attack: Stealthy and Persistent Attacks Against AI-IDE [64.47951172662745]
  Cuckoo Attack is a novel attack that achieves stealthy and persistent command execution by embedding malicious payloads into configuration files.<n>We formalize our attack paradigm into two stages, including initial infection and persistence.<n>We contribute seven actionable checkpoints for vendors to evaluate their product security.
  arXiv  Detail & Related papers  (2025-09-19T04:10:52Z)
• OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safety [58.201189860217724]
  We introduce OpenAgentSafety, a comprehensive framework for evaluating agent behavior across eight critical risk categories.<n>Unlike prior work, our framework evaluates agents that interact with real tools, including web browsers, code execution environments, file systems, bash shells, and messaging platforms.<n>It combines rule-based analysis with LLM-as-judge assessments to detect both overt and subtle unsafe behaviors.
  arXiv  Detail & Related papers  (2025-07-08T16:18:54Z)
• VPI-Bench: Visual Prompt Injection Attacks for Computer-Use Agents [74.6761188527948]
  Computer-Use Agents (CUAs) with full system access pose significant security and privacy risks.<n>We investigate Visual Prompt Injection (VPI) attacks, where malicious instructions are visually embedded within rendered user interfaces.<n>Our empirical study shows that current CUAs and BUAs can be deceived at rates of up to 51% and 100%, respectively, on certain platforms.
  arXiv  Detail & Related papers  (2025-06-03T05:21:50Z)
• The Hidden Dangers of Browsing AI Agents [0.0]
  This paper presents a comprehensive security evaluation of such agents, focusing on systemic vulnerabilities across multiple architectural layers.<n>Our work outlines the first end-to-end threat model for browsing agents and provides actionable guidance for securing their deployment in real-world environments.
  arXiv  Detail & Related papers  (2025-05-19T13:10:29Z)
• DoomArena: A framework for Testing AI Agents Against Evolving Security Threats [81.73540246946015]
  We present DoomArena, a security evaluation framework for AI agents.<n>It is a plug-in framework and integrates easily into realistic agentic frameworks.<n>It is modular and decouples the development of attacks from details of the environment in which the agent is deployed.
  arXiv  Detail & Related papers  (2025-04-18T20:36:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.