Automated Vulnerability Validation and Verification: A Large Language Model Approach

• URL: http://arxiv.org/abs/2509.24037v1
• Date: Sun, 28 Sep 2025 19:16:12 GMT
• Title: Automated Vulnerability Validation and Verification: A Large Language Model Approach
• Authors: Alireza Lotfi, Charalampos Katsis, Elisa Bertino,
• Abstract summary: This paper introduces an end-to-end multi-step pipeline leveraging generative AI, specifically large language models (LLMs)<n>Our approach extracts information from CVE disclosures in the National Vulnerability Database.<n>It augments it with external public knowledge (e.g., threat advisories, code snippets) using Retrieval-Augmented Generation (RAG)<n>The pipeline iteratively refines generated artifacts, validates attack success with test cases, and supports complex multi-container setups.
• Score: 7.482522010482827
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Software vulnerabilities remain a critical security challenge, providing entry points for attackers into enterprise networks. Despite advances in security practices, the lack of high-quality datasets capturing diverse exploit behavior limits effective vulnerability assessment and mitigation. This paper introduces an end-to-end multi-step pipeline leveraging generative AI, specifically large language models (LLMs), to address the challenges of orchestrating and reproducing attacks to known software vulnerabilities. Our approach extracts information from CVE disclosures in the National Vulnerability Database, augments it with external public knowledge (e.g., threat advisories, code snippets) using Retrieval-Augmented Generation (RAG), and automates the creation of containerized environments and exploit code for each vulnerability. The pipeline iteratively refines generated artifacts, validates attack success with test cases, and supports complex multi-container setups. Our methodology overcomes key obstacles, including noisy and incomplete vulnerability descriptions, by integrating LLMs and RAG to fill information gaps. We demonstrate the effectiveness of our pipeline across different vulnerability types, such as memory overflows, denial of service, and remote code execution, spanning diverse programming languages, libraries and years. In doing so, we uncover significant inconsistencies in CVE descriptions, emphasizing the need for more rigorous verification in the CVE disclosure process. Our approach is model-agnostic, working across multiple LLMs, and we open-source the artifacts to enable reproducibility and accelerate security research. To the best of our knowledge, this is the first system to systematically orchestrate and exploit known vulnerabilities in containerized environments by combining general-purpose LLM reasoning with CVE data and RAG-based context enrichment.

Related papers

• RealSec-bench: A Benchmark for Evaluating Secure Code Generation in Real-World Repositories [58.32028251925354]
  Large Language Models (LLMs) have demonstrated remarkable capabilities in code generation, but their proficiency in producing secure code remains a critical, under-explored area.<n>We introduce RealSec-bench, a new benchmark for secure code generation meticulously constructed from real-world, high-risk Java repositories.
  arXiv  Detail & Related papers  (2026-01-30T08:29:01Z)
• Just Ask: Curious Code Agents Reveal System Prompts in Frontier LLMs [65.6660735371212]
  We present textbftextscJustAsk, a framework that autonomously discovers effective extraction strategies through interaction alone.<n>It formulates extraction as an online exploration problem, using Upper Confidence Bound--based strategy selection and a hierarchical skill space spanning atomic probes and high-level orchestration.<n>Our results expose system prompts as a critical yet largely unprotected attack surface in modern agent systems.
  arXiv  Detail & Related papers  (2026-01-29T03:53:25Z)
• VulnLLM-R: Specialized Reasoning LLM with Agent Scaffold for Vulnerability Detection [45.69684471143409]
  VulnLLM-R is theemphfirst specialized reasoning LLM for vulnerability detection.<n>We train a reasoning model with seven billion parameters.<n>We show that VulnLLM-R has superior effectiveness and efficiency than SOTA static analysis tools.
  arXiv  Detail & Related papers  (2025-12-08T13:06:23Z)
• TypePilot: Leveraging the Scala Type System for Secure LLM-generated Code [46.747768845221735]
  Large language Models (LLMs) have shown remarkable proficiency in code generation tasks across various programming languages.<n>Their outputs often contain subtle but critical vulnerabilities, posing significant risks when deployed in security-sensitive or mission-critical systems.<n>This paper introduces TypePilot, an agentic AI framework designed to enhance the security and robustness of LLM-generated code.
  arXiv  Detail & Related papers  (2025-10-13T08:44:01Z)
• VulnRepairEval: An Exploit-Based Evaluation Framework for Assessing Large Language Model Vulnerability Repair Capabilities [41.85494398578654]
  VulnRepairEval is an evaluation framework anchored in functional Proof-of-Concept exploits.<n>Our framework delivers a comprehensive, containerized evaluation pipeline that enables reproducible differential assessment.
  arXiv  Detail & Related papers  (2025-09-03T14:06:10Z)
• Secure Tug-of-War (SecTOW): Iterative Defense-Attack Training with Reinforcement Learning for Multimodal Model Security [63.41350337821108]
  We propose Secure Tug-of-War (SecTOW) to enhance the security of multimodal large language models (MLLMs)<n>SecTOW consists of two modules: a defender and an auxiliary attacker, both trained iteratively using reinforcement learning (GRPO)<n>We show that SecTOW significantly improves security while preserving general performance.
  arXiv  Detail & Related papers  (2025-07-29T17:39:48Z)
• Improving LLM Reasoning for Vulnerability Detection via Group Relative Policy Optimization [45.799380822683034]
  We present an extensive study aimed at advancing RL-based finetuning techniques for Large Language Models (LLMs)<n>We highlight key limitations of commonly adopted LLMs, such as their tendency to over-predict certain types of vulnerabilities while failing to detect others.<n>To address this challenge, we explore the use of Group Relative Policy Optimization (GRPO), a recent policy-gradient method, for guiding LLM behavior through structured, rule-based rewards.
  arXiv  Detail & Related papers  (2025-07-03T11:52:45Z)
• CyberGym: Evaluating AI Agents' Cybersecurity Capabilities with Real-World Vulnerabilities at Scale [46.76144797837242]
  Large language model (LLM) agents are becoming increasingly skilled at handling cybersecurity tasks autonomously.<n>Existing benchmarks fall short, often failing to capture real-world scenarios or being limited in scope.<n>We introduce CyberGym, a large-scale and high-quality cybersecurity evaluation framework featuring 1,507 real-world vulnerabilities.
  arXiv  Detail & Related papers  (2025-06-03T07:35:14Z)
• VulZoo: A Comprehensive Vulnerability Intelligence Dataset [12.229092589037808]
  VulZoo is a comprehensive vulnerability intelligence dataset that covers 17 popular vulnerability information sources. We make VulZoo publicly available and maintain it with incremental updates to facilitate future research.
  arXiv  Detail & Related papers  (2024-06-24T06:39:07Z)
• Towards Explainable Vulnerability Detection with Large Language Models [14.243344783348398]
  Software vulnerabilities pose significant risks to the security and integrity of software systems.<n>The advent of large language models (LLMs) has introduced transformative potential due to their advanced generative capabilities.<n>In this paper, we propose LLMVulExp, an automated framework designed to specialize LLMs for the dual tasks of vulnerability detection and explanation.
  arXiv  Detail & Related papers  (2024-06-14T04:01:25Z)
• On Security Weaknesses and Vulnerabilities in Deep Learning Systems [32.14068820256729]
  We specifically look into deep learning (DL) framework and perform the first systematic study of vulnerabilities in DL systems. We propose a two-stream data analysis framework to explore vulnerability patterns from various databases. We conducted a large-scale empirical study of 3,049 DL vulnerabilities to better understand the patterns of vulnerability and the challenges in fixing them.
  arXiv  Detail & Related papers  (2024-06-12T23:04:13Z)
• How Far Have We Gone in Vulnerability Detection Using Large Language Models [15.09461331135668]
  We introduce a comprehensive vulnerability benchmark VulBench. This benchmark aggregates high-quality data from a wide range of CTF challenges and real-world applications. We find that several LLMs outperform traditional deep learning approaches in vulnerability detection.
  arXiv  Detail & Related papers  (2023-11-21T08:20:39Z)
• Enhancing Large Language Models for Secure Code Generation: A Dataset-driven Study on Vulnerability Mitigation [24.668682498171776]
  Large language models (LLMs) have brought significant advancements to code generation, benefiting both novice and experienced developers. However, their training using unsanitized data from open-source repositories, like GitHub, introduces the risk of inadvertently propagating security vulnerabilities. This paper presents a comprehensive study focused on evaluating and enhancing code LLMs from a software security perspective.
  arXiv  Detail & Related papers  (2023-10-25T00:32:56Z)
• REEF: A Framework for Collecting Real-World Vulnerabilities and Fixes [40.401211102969356]
  We propose an automated collecting framework REEF to collect REal-world vulnErabilities and Fixes from open-source repositories. We develop a multi-language crawler to collect vulnerabilities and their fixes, and design metrics to filter for high-quality vulnerability-fix pairs. Through extensive experiments, we demonstrate that our approach can collect high-quality vulnerability-fix pairs and generate strong explanations.
  arXiv  Detail & Related papers  (2023-09-15T02:50:08Z)
• VELVET: a noVel Ensemble Learning approach to automatically locate VulnErable sTatements [62.93814803258067]
  This paper presents VELVET, a novel ensemble learning approach to locate vulnerable statements in source code. Our model combines graph-based and sequence-based neural networks to successfully capture the local and global context of a program graph. VELVET achieves 99.6% and 43.6% top-1 accuracy over synthetic data and real-world data, respectively.
  arXiv  Detail & Related papers  (2021-12-20T22:45:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.