Towards Automated Pentesting with Large Language Models

• URL: http://arxiv.org/abs/2604.11772v1
• Date: Mon, 13 Apr 2026 17:41:51 GMT
• Title: Towards Automated Pentesting with Large Language Models
• Abstract summary: Large Language Models (LLMs) are redefining offensive cybersecurity by allowing the generation of harmful machine code with minimal human intervention.<n>We present RedShell, a framework that generates malicious code targeting Microsoft Windows vulnerabilities.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) are redefining offensive cybersecurity by allowing the generation of harmful machine code with minimal human intervention. While attackers take advantage of dark LLMs such as XXXGPT and WolfGPT to produce malicious code, ethical hackers can follow similar approaches to automate traditional pentesting workflows. In this work, we present RedShell, a privacy-preserving, hardware-efficient framework that leverages fine-tuned LLMs to assist pentesters in generating offensive PowerShell code targeting Microsoft Windows vulnerabilities. RedShell was trained on a malicious PowerShell dataset from the literature, which we further enhanced with manually curated code samples. Experiments show that our framework achieves over 90% syntactic validity in generated samples and strong semantic alignment with reference pentesting snippets, outperforming state-of-the-art counterparts in distance metrics such as edit distance (above 50% average code similarity). Additionally, functional experiments emphasize the execution reliability of the snippets produced by RedShell in a testing scenario that mirrors real-world settings. This work sheds light on the state-of-the-art research in the field of Generative AI applied to malicious code generation and automated testing, acknowledging the potential benefits that LLMs hold within controlled environments such as pentesting.

Related papers

• Beyond the Syntax: Do Security Experts Trust LLMs for NIDS Rule Engineering? [70.17374586573033]
  This paper presents a human-centered investigation into Large Language Models (LLMs)-based NIDS rule engineering.<n>Our evaluation reveals a syntax-semantics paradox: although LLMs generate syntactically correct rules, experts find them only partially deployable.<n>While large-scale models consistently produce syntactically valid rules, small models are largely ineffective for IDS rule generation.
  arXiv  Detail & Related papers  (2026-07-07T07:09:56Z)
• AI-Generated PowerShell Malware: An Experimental Framework and Dataset [3.315877207045108]
  Generative AI has emerged as a significant cybersecurity threat.<n>We present a novel, manually curated dataset of real-world malware annotated in natural language.<n>Our results reveal a high degree of similarity between real malware and LLM-generated ones in terms of triggered OS malicious events.
  arXiv  Detail & Related papers  (2026-06-29T18:46:21Z)
• RedShell: A Generative AI-Based Approach to Ethical Hacking [0.0]
  We propose RedShell, a tool that allows ethical hackers to generate malicious code.<n>Our experiments demonstrate the strong capabilities of RedShell in generating syntactically valid, with fewer than 10% of the generated samples resulting in parse errors.<n>This work sheds light on the state-of-the-art research in the field of Generative AI applied to Pentesting, and also serves as a steppingstone for future advancements.
  arXiv  Detail & Related papers  (2026-04-13T14:10:44Z)
• Detecting PowerShell-based Fileless Cryptojacking Attacks Using Machine Learning [0.09320657506524148]
  cryptojacking attacks have become effective with stealthy techniques based on Syn-based exploitation in Windows OS environments.<n> Abstracttax Tree (AST)-based fine-tuned CodeBERT achieved a high recall rate.
  arXiv  Detail & Related papers  (2026-02-20T15:32:15Z)
• RealSec-bench: A Benchmark for Evaluating Secure Code Generation in Real-World Repositories [58.32028251925354]
  Large Language Models (LLMs) have demonstrated remarkable capabilities in code generation, but their proficiency in producing secure code remains a critical, under-explored area.<n>We introduce RealSec-bench, a new benchmark for secure code generation meticulously constructed from real-world, high-risk Java repositories.
  arXiv  Detail & Related papers  (2026-01-30T08:29:01Z)
• Lightweight Yet Secure: Secure Scripting Language Generation via Lightweight LLMs [14.355160053196824]
  Large language models (LLMs) have demonstrated strong capabilities in code generation, vulnerability detection, and automated repair for languages like Python and JavaScript.<n>We present SecGenEval-PS, a benchmark designed to evaluate LLMs on secure scripting generation, security analysis, and automated repair.<n>We propose PSSec, a framework that combines data synthesis with fine-tuning to enhance model security capabilities.
  arXiv  Detail & Related papers  (2026-01-10T04:00:56Z)
• Trust Me, I Know This Function: Hijacking LLM Static Analysis using Bias [3.178301843099705]
  Large Language Models (LLMs) are increasingly trusted to perform automated code review and static analysis at scale.<n>This paper identifies and exploit a critical vulnerability in LLM-based code analysis.<n>We develop a fully automated, black-box algorithm that discovers and injects Familiar Pattern Attack (FPA) into target code.
  arXiv  Detail & Related papers  (2025-08-24T13:42:48Z)
• ObscuraCoder: Powering Efficient Code LM Pre-Training Via Obfuscation Grounding [60.37988508851391]
  Language models (LMs) have become a staple of the code-writing toolbox.<n>Research exploring modifications to Code-LMs' pre-training objectives, geared towards improving data efficiency and better disentangling between syntax and semantics, has been noticeably sparse.<n>In this work, we examine grounding on obfuscated code as a means of helping Code-LMs look beyond the surface-form syntax and enhance their pre-training sample efficiency.
  arXiv  Detail & Related papers  (2025-03-27T23:08:53Z)
• AutoPT: How Far Are We from the End2End Automated Web Penetration Testing? [54.65079443902714]
  We introduce AutoPT, an automated penetration testing agent based on the principle of PSM driven by LLMs. Our results show that AutoPT outperforms the baseline framework ReAct on the GPT-4o mini model.
  arXiv  Detail & Related papers  (2024-11-02T13:24:30Z)
• ShadowCode: Towards (Automatic) External Prompt Injection Attack against Code LLMs [56.46702494338318]
  This paper introduces a new attack paradigm: (automatic) external prompt injection against code-oriented large language models.<n>We propose ShadowCode, a simple yet effective method that automatically generates induced perturbations based on code simulation.<n>We evaluate our method across 13 distinct malicious objectives, generating 31 threat cases spanning three popular programming languages.
  arXiv  Detail & Related papers  (2024-07-12T10:59:32Z)
• LLMs as Hackers: Autonomous Linux Privilege Escalation Attacks [0.0]
  We introduce hackingBuddyGPT, a fully automated Language Models (LLMs)-driven prototype for autonomous Linux privilege-escalation attacks.<n>We show that GPT-4-Turbo demonstrates high efficacy, successfully exploiting 33-83% of vulnerabilities, a performance comparable to human pen-testers.
  arXiv  Detail & Related papers  (2023-10-17T17:15:41Z)
• AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models [54.95912006700379]
  We introduce AutoDAN, a novel jailbreak attack against aligned Large Language Models. AutoDAN can automatically generate stealthy jailbreak prompts by the carefully designed hierarchical genetic algorithm.
  arXiv  Detail & Related papers  (2023-10-03T19:44:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.