RapidPen: Fully Automated IP-to-Shell Penetration Testing with LLM-based Agents

• URL: http://arxiv.org/abs/2502.16730v1
• Date: Sun, 23 Feb 2025 21:57:46 GMT
• Title: RapidPen: Fully Automated IP-to-Shell Penetration Testing with LLM-based Agents
• Authors: Sho Nakatani,
• Abstract summary: RapidPen is a fully automated penetration testing framework.<n>It autonomously discovers and exploit vulnerabilities, starting from a single IP address.<n>It achieved shell access within 200-400 seconds at a per-run cost of approximately $0.3-$0.6.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: We present RapidPen, a fully automated penetration testing (pentesting) framework that addresses the challenge of achieving an initial foothold (IP-to-Shell) without human intervention. Unlike prior approaches that focus primarily on post-exploitation or require a human-in-the-loop, RapidPen leverages large language models (LLMs) to autonomously discover and exploit vulnerabilities, starting from a single IP address. By integrating advanced ReAct-style task planning (Re) with retrieval-augmented knowledge bases of successful exploits, along with a command-generation and direct execution feedback loop (Act), RapidPen systematically scans services, identifies viable attack vectors, and executes targeted exploits in a fully automated manner. In our evaluation against a vulnerable target from the Hack The Box platform, RapidPen achieved shell access within 200-400 seconds at a per-run cost of approximately \$0.3-\$0.6, demonstrating a 60\% success rate when reusing prior "success-case" data. These results underscore the potential of truly autonomous pentesting for both security novices and seasoned professionals. Organizations without dedicated security teams can leverage RapidPen to quickly identify critical vulnerabilities, while expert pentesters can offload repetitive tasks and focus on complex challenges. Ultimately, our work aims to make penetration testing more accessible and cost-efficient, thereby enhancing the overall security posture of modern software ecosystems.

Related papers

• PenForge: On-the-Fly Expert Agent Construction for Automated Penetration Testing [13.699270001160302]
  PenForge is a framework that dynamically constructs expert agents during testing rather than relying on those prepared beforehand.<n>It achieves a 30.0% exploit success rate on CVE-Bench in the particularly challenging zero-day setting, which is a 3 times improvement over the state-of-the-art.
  arXiv  Detail & Related papers  (2026-01-11T13:29:32Z)
• Shell or Nothing: Real-World Benchmarks and Memory-Activated Agents for Automated Penetration Testing [23.554239007767276]
  We introduce the first real-world, agent-oriented pentesting benchmark, TermiBench.<n>We propose TermiAgent, a multi-agent penetration testing framework.<n>In evaluations, our work outperforms state-of-the-art agents, exhibiting stronger penetration testing capability.
  arXiv  Detail & Related papers  (2025-09-11T07:30:44Z)
• Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competition [101.86739402748995]
  We run the largest public red-teaming competition to date, targeting 22 frontier AI agents across 44 realistic deployment scenarios.<n>We build the Agent Red Teaming benchmark and evaluate it across 19 state-of-the-art models.<n>Our findings highlight critical and persistent vulnerabilities in today's AI agents.
  arXiv  Detail & Related papers  (2025-07-28T05:13:04Z)
• OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safety [58.201189860217724]
  We introduce OpenAgentSafety, a comprehensive framework for evaluating agent behavior across eight critical risk categories.<n>Unlike prior work, our framework evaluates agents that interact with real tools, including web browsers, code execution environments, file systems, bash shells, and messaging platforms.<n>It combines rule-based analysis with LLM-as-judge assessments to detect both overt and subtle unsafe behaviors.
  arXiv  Detail & Related papers  (2025-07-08T16:18:54Z)
• AgentVigil: Generic Black-Box Red-teaming for Indirect Prompt Injection against LLM Agents [54.29555239363013]
  We propose a generic black-box fuzzing framework, AgentVigil, to automatically discover and exploit indirect prompt injection vulnerabilities.<n>We evaluate AgentVigil on two public benchmarks, AgentDojo and VWA-adv, where it achieves 71% and 70% success rates against agents based on o3-mini and GPT-4o.<n>We apply our attacks in real-world environments, successfully misleading agents to navigate to arbitrary URLs, including malicious sites.
  arXiv  Detail & Related papers  (2025-05-09T07:40:17Z)
• WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks [36.97842000562324]
  We introduce WASP -- a new benchmark for end-to-end evaluation of Web Agent Security against Prompt injection attacks.<n>We show that even top-tier AI models, including those with advanced reasoning capabilities, can be deceived by simple, low-effort human-written injections.<n>Our end-to-end evaluation reveals a previously unobserved insight: while attacks partially succeed in up to 86% of the case, even state-of-the-art agents often struggle to fully complete the attacker goals.
  arXiv  Detail & Related papers  (2025-04-22T17:51:03Z)
• Auto-RT: Automatic Jailbreak Strategy Exploration for Red-Teaming Large Language Models [62.12822290276912]
  Auto-RT is a reinforcement learning framework that automatically explores and optimize complex attack strategies.<n>By significantly improving exploration efficiency and automatically optimizing attack strategies, Auto-RT detects a boarder range of vulnerabilities, achieving a faster detection speed and 16.63% higher success rates compared to existing methods.
  arXiv  Detail & Related papers  (2025-01-03T14:30:14Z)
• AutoPT: How Far Are We from the End2End Automated Web Penetration Testing? [54.65079443902714]
  We introduce AutoPT, an automated penetration testing agent based on the principle of PSM driven by LLMs. Our results show that AutoPT outperforms the baseline framework ReAct on the GPT-4o mini model.
  arXiv  Detail & Related papers  (2024-11-02T13:24:30Z)
• AutoPenBench: Benchmarking Generative Agents for Penetration Testing [42.681170697805726]
  This paper introduces AutoPenBench, an open benchmark for evaluating generative agents in automated penetration testing. We present a comprehensive framework that includes 33 tasks, each representing a vulnerable system that the agent has to attack. We show the benefits of AutoPenBench by testing two agent architectures: a fully autonomous and a semi-autonomous supporting human interaction.
  arXiv  Detail & Related papers  (2024-10-04T08:24:15Z)
• ExACT: Teaching AI Agents to Explore with Reflective-MCTS and Exploratory Learning [78.42927884000673]
  ExACT is an approach to combine test-time search and self-learning to build o1-like models for agentic applications.<n>We first introduce Reflective Monte Carlo Tree Search (R-MCTS), a novel test time algorithm designed to enhance AI agents' ability to explore decision space on the fly.<n>Next, we introduce Exploratory Learning, a novel learning strategy to teach agents to search at inference time without relying on any external search algorithms.
  arXiv  Detail & Related papers  (2024-10-02T21:42:35Z)
• Hacking, The Lazy Way: LLM Augmented Pentesting [0.0]
  "LLM Augmented Pentesting" is demonstrated through a tool named "Pentest Copilot" Our research includes a "chain of thought" mechanism to streamline token usage and boost performance. We propose a novel file analysis approach, enabling LLMs to understand files.
  arXiv  Detail & Related papers  (2024-09-14T17:40:35Z)
• BreachSeek: A Multi-Agent Automated Penetration Tester [0.0]
  BreachSeek is an AI-driven multi-agent software platform that identifies and exploits vulnerabilities without human intervention. In preliminary evaluations, BreachSeek successfully exploited vulnerabilities in exploitable machines within local networks. Future developments aim to expand its capabilities, positioning it as an indispensable tool for cybersecurity professionals.
  arXiv  Detail & Related papers  (2024-08-31T19:15:38Z)
• Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models [79.0183835295533]
  We introduce the first benchmark for indirect prompt injection attacks, named BIPIA, to assess the risk of such vulnerabilities.<n>Our analysis identifies two key factors contributing to their success: LLMs' inability to distinguish between informational context and actionable instructions, and their lack of awareness in avoiding the execution of instructions within external content.<n>We propose two novel defense mechanisms-boundary awareness and explicit reminder-to address these vulnerabilities in both black-box and white-box settings.
  arXiv  Detail & Related papers  (2023-12-21T01:08:39Z)
• Raij\=u: Reinforcement Learning-Guided Post-Exploitation for Automating Security Assessment of Network Systems [0.0]
  Raij=u framework is a Reinforcement Learning-driven automation approach. We implement two RL algorithms to train specialized agents capable of making intelligent actions. Agents achieve over 84% of successful attacks with under 55 attack steps given.
  arXiv  Detail & Related papers  (2023-09-27T09:36:22Z)
• Adversarial EXEmples: A Survey and Experimental Evaluation of Practical Attacks on Machine Learning for Windows Malware Detection [67.53296659361598]
  adversarial EXEmples can bypass machine learning-based detection by perturbing relatively few input bytes. We develop a unifying framework that does not only encompass and generalize previous attacks against machine-learning models, but also includes three novel attacks. These attacks, named Full DOS, Extend and Shift, inject the adversarial payload by respectively manipulating the DOS header, extending it, and shifting the content of the first section.
  arXiv  Detail & Related papers  (2020-08-17T07:16:57Z)
• Adversarial vs behavioural-based defensive AI with joint, continual and active learning: automated evaluation of robustness to deception, poisoning and concept drift [62.997667081978825]
  Recent advancements in Artificial Intelligence (AI) have brought new capabilities to behavioural analysis (UEBA) for cyber-security. In this paper, we present a solution to effectively mitigate this attack by improving the detection process and efficiently leveraging human expertise.
  arXiv  Detail & Related papers  (2020-01-13T13:54:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.