HackWorld: Evaluating Computer-Use Agents on Exploiting Web Application Vulnerabilities

• URL: http://arxiv.org/abs/2510.12200v1
• Date: Tue, 14 Oct 2025 06:52:15 GMT
• Title: HackWorld: Evaluating Computer-Use Agents on Exploiting Web Application Vulnerabilities
• Authors: Xiaoxue Ren, Penghao Jiang, Kaixin Li, Zhiyong Huang, Xiaoning Du, Jiaojiao Jiang, Zhenchang Xing, Jiamou Sun, Terry Yue Zhuo,
• Abstract summary: HackWorld is the first framework for evaluating computer-use agents' capabilities to exploit web application vulnerabilities via visual interaction.<n>It includes 36 real-world applications across 11 frameworks and 7 languages, featuring realistic flaws such as injection vulnerabilities, authentication bypasses, and unsafe input handling.<n>It tests CUAs' capacity to identify and exploit these weaknesses while navigating complex web interfaces.
• Score: 20.201614123811872
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Web applications are prime targets for cyberattacks as gateways to critical services and sensitive data. Traditional penetration testing is costly and expertise-intensive, making it difficult to scale with the growing web ecosystem. While language model agents show promise in cybersecurity, modern web applications demand visual understanding, dynamic content handling, and multi-step interactions that only computer-use agents (CUAs) can perform. Yet, their ability to discover and exploit vulnerabilities through graphical interfaces remains largely unexplored. We present HackWorld, the first framework for systematically evaluating CUAs' capabilities to exploit web application vulnerabilities via visual interaction. Unlike sanitized benchmarks, HackWorld includes 36 real-world applications across 11 frameworks and 7 languages, featuring realistic flaws such as injection vulnerabilities, authentication bypasses, and unsafe input handling. Using a Capture-the-Flag (CTF) setup, it tests CUAs' capacity to identify and exploit these weaknesses while navigating complex web interfaces. Evaluation of state-of-the-art CUAs reveals concerning trends: exploitation rates below 12% and low cybersecurity awareness. CUAs often fail at multi-step attack planning and misuse security tools. These results expose the current limitations of CUAs in web security contexts and highlight opportunities for developing more security-aware agents capable of effective vulnerability detection and exploitation.

Related papers

• Just Ask: Curious Code Agents Reveal System Prompts in Frontier LLMs [65.6660735371212]
  We present textbftextscJustAsk, a framework that autonomously discovers effective extraction strategies through interaction alone.<n>It formulates extraction as an online exploration problem, using Upper Confidence Bound--based strategy selection and a hierarchical skill space spanning atomic probes and high-level orchestration.<n>Our results expose system prompts as a critical yet largely unprotected attack surface in modern agent systems.
  arXiv  Detail & Related papers  (2026-01-29T03:53:25Z)
• CaMeLs Can Use Computers Too: System-level Security for Computer Use Agents [60.98294016925157]
  AI agents are vulnerable to prompt injection attacks, where malicious content hijacks agent behavior to steal credentials or cause financial loss.<n>We introduce Single-Shot Planning for CUAs, where a trusted planner generates a complete execution graph with conditional branches before any observation of potentially malicious content.<n>Although this architectural isolation successfully prevents instruction injections, we show that additional measures are needed to prevent Branch Steering attacks.
  arXiv  Detail & Related papers  (2026-01-14T23:06:35Z)
• LLM Agents for Automated Web Vulnerability Reproduction: Are We There Yet? [9.817896112083647]
  Large language model (LLM) agents have demonstrated remarkable capabilities in software engineering and cybersecurity tasks.<n>Recent advances suggest promising potential, but challenges remain in applying LLM agents to real-world web vulnerability reproduction scenarios.<n>This paper presents the first comprehensive evaluation of state-of-the-art LLM agents for automated web vulnerability reproduction.
  arXiv  Detail & Related papers  (2025-10-16T14:04:46Z)
• Code Agent can be an End-to-end System Hacker: Benchmarking Real-world Threats of Computer-use Agent [64.08182031659047]
  We propose AdvCUA, the first benchmark aligned with real-world TTPs in MITRE ATT&CK Enterprise Matrix.<n>We evaluate the existing five mainstream CUAs, including ReAct, AutoGPT, Gemini CLI, and Cursor CLI.<n>Results demonstrate that current frontier CUAs do not adequately cover OS security-centric threats.
  arXiv  Detail & Related papers  (2025-10-08T03:35:23Z)
• A Systematization of Security Vulnerabilities in Computer Use Agents [1.3560089220432787]
  We conduct a systematic threat analysis and testing of real-world CUAs under adversarial conditions.<n>We identify seven classes of risks unique to the CUA paradigm, and analyze three concrete exploit scenarios in depth.<n>These case studies reveal deeper architectural flaws across current CUA implementations.
  arXiv  Detail & Related papers  (2025-07-07T19:50:21Z)
• VPI-Bench: Visual Prompt Injection Attacks for Computer-Use Agents [74.6761188527948]
  Computer-Use Agents (CUAs) with full system access pose significant security and privacy risks.<n>We investigate Visual Prompt Injection (VPI) attacks, where malicious instructions are visually embedded within rendered user interfaces.<n>Our empirical study shows that current CUAs and BUAs can be deceived at rates of up to 51% and 100%, respectively, on certain platforms.
  arXiv  Detail & Related papers  (2025-06-03T05:21:50Z)
• Towards Trustworthy GUI Agents: A Survey [64.6445117343499]
  This survey examines the trustworthiness of GUI agents in five critical dimensions.<n>We identify major challenges such as vulnerability to adversarial attacks, cascading failure modes in sequential decision-making.<n>As GUI agents become more widespread, establishing robust safety standards and responsible development practices is essential.
  arXiv  Detail & Related papers  (2025-03-30T13:26:00Z)
• CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities [6.752938800468733]
  Large language model (LLM) agents are increasingly capable of autonomously conducting cyberattacks.<n>Existing benchmarks fall short as they are limited to abstracted Capture the Flag competitions or lack comprehensive coverage.<n>We introduce CVE-Bench, a real-world cybersecurity benchmark based on critical-severity Common Vulnerabilities and Exposures.
  arXiv  Detail & Related papers  (2025-03-21T17:32:32Z)
• Adding web pentesting functionality to PTHelper [0.4779196219827506]
  This project is the direct continuation of the previous initiative called PThelper: An open source tool to support the Penetration Testing process. This continuation is focused on expanding PThelper with the functionality to detect and later report web vulnerabilities.
  arXiv  Detail & Related papers  (2024-10-16T10:05:56Z)
• BreachSeek: A Multi-Agent Automated Penetration Tester [0.0]
  BreachSeek is an AI-driven multi-agent software platform that identifies and exploits vulnerabilities without human intervention. In preliminary evaluations, BreachSeek successfully exploited vulnerabilities in exploitable machines within local networks. Future developments aim to expand its capabilities, positioning it as an indispensable tool for cybersecurity professionals.
  arXiv  Detail & Related papers  (2024-08-31T19:15:38Z)
• Rethinking the Vulnerabilities of Face Recognition Systems:From a Practical Perspective [53.24281798458074]
  Face Recognition Systems (FRS) have increasingly integrated into critical applications, including surveillance and user authentication. Recent studies have revealed vulnerabilities in FRS to adversarial (e.g., adversarial patch attacks) and backdoor attacks (e.g., training data poisoning)
  arXiv  Detail & Related papers  (2024-05-21T13:34:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.