Construction and Evaluation of LLM-based agents for Semi-Autonomous penetration testing

• URL: http://arxiv.org/abs/2502.15506v1
• Date: Fri, 21 Feb 2025 15:02:39 GMT
• Title: Construction and Evaluation of LLM-based agents for Semi-Autonomous penetration testing
• Authors: Masaya Kobayashi, Masane Fuchi, Amar Zanashir, Tomonori Yoneda, Tomohiro Takagi,
• Abstract summary: High-performance large language models (LLMs) have advanced across various domains.<n>In highly specialized fields such as cybersecurity, full autonomy remains a challenge.<n>We propose a system that semi-autonomously executes complex cybersecurity by employing multiple LLMs modules.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the emergence of high-performance large language models (LLMs) such as GPT, Claude, and Gemini, the autonomous and semi-autonomous execution of tasks has significantly advanced across various domains. However, in highly specialized fields such as cybersecurity, full autonomy remains a challenge. This difficulty primarily stems from the limitations of LLMs in reasoning capabilities and domain-specific knowledge. We propose a system that semi-autonomously executes complex cybersecurity workflows by employing multiple LLMs modules to formulate attack strategies, generate commands, and analyze results, thereby addressing the aforementioned challenges. In our experiments using Hack The Box virtual machines, we confirmed that our system can autonomously construct attack strategies, issue appropriate commands, and automate certain processes, thereby reducing the need for manual intervention.

Related papers

• ATAG: AI-Agent Application Threat Assessment with Attack Graphs [23.757154032523093]
  This paper introduces AI-agent application Threat assessment with Attack Graphs (ATAG)<n>ATAG is a novel framework designed to systematically analyze the security risks associated with AI-agent applications.<n>It facilitates proactive identification and mitigation of AI-agent threats in multi-agent applications.
  arXiv  Detail & Related papers  (2025-06-03T13:25:40Z)
• Benchmarking LLMs in an Embodied Environment for Blue Team Threat Hlunting [14.810934670172479]
  Large Language Models (LLMs) offer promising capabilities for enhancing threat analysis.<n>Their effectiveness in real-world blue team threat-hunting scenarios remains insufficiently explored.<n>We present CYBERTEAM, a benchmark designed to guide LLMs in blue teaming practice.
  arXiv  Detail & Related papers  (2025-05-17T08:33:50Z)
• AgentVigil: Generic Black-Box Red-teaming for Indirect Prompt Injection against LLM Agents [54.29555239363013]
  We propose a generic black-box fuzzing framework, AgentVigil, to automatically discover and exploit indirect prompt injection vulnerabilities.<n>We evaluate AgentVigil on two public benchmarks, AgentDojo and VWA-adv, where it achieves 71% and 70% success rates against agents based on o3-mini and GPT-4o.<n>We apply our attacks in real-world environments, successfully misleading agents to navigate to arbitrary URLs, including malicious sites.
  arXiv  Detail & Related papers  (2025-05-09T07:40:17Z)
• Les Dissonances: Cross-Tool Harvesting and Polluting in Multi-Tool Empowered LLM Agents [15.15485816037418]
  This paper presents the first systematic security analysis of task control flows in multi-tool-enabled LLM agents. We identify a novel threat, Cross-Tool Harvesting and Polluting (XTHP), which includes multiple attack vectors. To understand the impact of this threat, we developed Chord, a dynamic scanning tool designed to automatically detect real-world agent tools susceptible to XTHP attacks.
  arXiv  Detail & Related papers  (2025-04-04T01:41:06Z)
• Scaling Autonomous Agents via Automatic Reward Modeling And Planning [52.39395405893965]
  Large language models (LLMs) have demonstrated remarkable capabilities across a range of tasks.<n>However, they still struggle with problems requiring multi-step decision-making and environmental feedback.<n>We propose a framework that can automatically learn a reward model from the environment without human annotations.
  arXiv  Detail & Related papers  (2025-02-17T18:49:25Z)
• Can LLMs Hack Enterprise Networks? Autonomous Assumed Breach Penetration-Testing Active Directory Networks [3.11537581064266]
  This paper investigates the feasibility and effectiveness of using Large Language Model (LLM)-driven autonomous systems to address challenges in real-world Microsoft Active Directory (AD) enterprise networks.<n>Our prototype, cochise, represents the first demonstration of a fully autonomous, LLM-driven framework capable of compromising accounts within a real-life Microsoft AD testbed (GOAD)<n>Key findings highlight their ability to dynamically adapt attack strategies, perform inter-context attacks, and generate scenario-specific attack parameters.
  arXiv  Detail & Related papers  (2025-02-06T17:12:43Z)
• MALMM: Multi-Agent Large Language Models for Zero-Shot Robotics Manipulation [52.739500459903724]
  Large Language Models (LLMs) have demonstrated remarkable planning abilities across various domains, including robotics manipulation and navigation. We propose a novel multi-agent LLM framework that distributes high-level planning and low-level control code generation across specialized LLM agents. We evaluate our approach on nine RLBench tasks, including long-horizon tasks, and demonstrate its ability to solve robotics manipulation in a zero-shot setting.
  arXiv  Detail & Related papers  (2024-11-26T17:53:44Z)
• PentestAgent: Incorporating LLM Agents to Automated Penetration Testing [6.815381197173165]
  Manual penetration testing is time-consuming and expensive. Recent advancements in large language models (LLMs) offer new opportunities for enhancing penetration testing. We propose PentestAgent, a novel LLM-based automated penetration testing framework.
  arXiv  Detail & Related papers  (2024-11-07T21:10:39Z)
• AutoPT: How Far Are We from the End2End Automated Web Penetration Testing? [54.65079443902714]
  We introduce AutoPT, an automated penetration testing agent based on the principle of PSM driven by LLMs. Our results show that AutoPT outperforms the baseline framework ReAct on the GPT-4o mini model.
  arXiv  Detail & Related papers  (2024-11-02T13:24:30Z)
• AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoML [56.565200973244146]
  Automated machine learning (AutoML) accelerates AI development by automating tasks in the development pipeline. Recent works have started exploiting large language models (LLM) to lessen such burden. This paper proposes AutoML-Agent, a novel multi-agent framework tailored for full-pipeline AutoML.
  arXiv  Detail & Related papers  (2024-10-03T20:01:09Z)
• From Sands to Mansions: Simulating Full Attack Chain with LLM-Organized Knowledge [10.065241604400223]
  Multi-stage attack simulations offer a promising approach to enhance system evaluation efficiency.<n> simulating a full attack chain is complex and requires significant time and expertise from security professionals.<n>We introduce Aurora, a system that autonomously simulates full attack chains based on external attack tools and threat intelligence reports.
  arXiv  Detail & Related papers  (2024-07-24T01:33:57Z)
• Can We Trust Embodied Agents? Exploring Backdoor Attacks against Embodied LLM-based Decision-Making Systems [27.316115171846953]
  Large Language Models (LLMs) have shown significant promise in real-world decision-making tasks for embodied AI. LLMs are fine-tuned to leverage their inherent common sense and reasoning abilities while being tailored to specific applications. This fine-tuning process introduces considerable safety and security vulnerabilities, especially in safety-critical cyber-physical systems.
  arXiv  Detail & Related papers  (2024-05-27T17:59:43Z)
• TaskBench: Benchmarking Large Language Models for Task Automation [82.2932794189585]
  We introduce TaskBench, a framework to evaluate the capability of large language models (LLMs) in task automation. Specifically, task decomposition, tool selection, and parameter prediction are assessed. Our approach combines automated construction with rigorous human verification, ensuring high consistency with human evaluation.
  arXiv  Detail & Related papers  (2023-11-30T18:02:44Z)
• LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving [87.1164964709168]
  This work employs Large Language Models (LLMs) as a decision-making component for complex autonomous driving scenarios. Extensive experiments demonstrate that our proposed method not only consistently surpasses baseline approaches in single-vehicle tasks, but also helps handle complex driving behaviors even multi-vehicle coordination.
  arXiv  Detail & Related papers  (2023-10-04T17:59:49Z)
• Automating Cyber Threat Hunting Using NLP, Automated Query Generation, and Genetic Perturbation [8.669461942767098]
  We have developed the WILEE system that cyber threat hunting by translating high-level threat descriptions into many possible concrete implementations. Both the (high-level) abstract and (low-level) concrete implementations are represented using a custom domain specific language. WILEE uses the implementations along with other logic, also written in the DSL, to automatically generate queries to confirm (or refute) any hypotheses tied to the potential adversarial.
  arXiv  Detail & Related papers  (2021-04-23T13:19:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.