Revisiting Vulnerability Patch Identification on Data in the Wild

• URL: http://arxiv.org/abs/2603.17266v1
• Date: Wed, 18 Mar 2026 01:45:39 GMT
• Title: Revisiting Vulnerability Patch Identification on Data in the Wild
• Abstract summary: Security patch detectors commonly trained and evaluated on security patches linked from vulnerability reports in the National Vulnerability Database (NVD)<n>We show that models trained on NVD-derived data show substantially decreased performance, with decreases in F1-score of up to 90% when tested on in-the-wild security patches.<n>We find that constructing a dataset that combines security patches from NVD data with a small subset of manually identified security patches can improve model robustness.
• Score: 12.263176392249745
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Attacks can exploit zero-day or one-day vulnerabilities that are not publicly disclosed. To detect these vulnerabilities, security researchers monitor development activities in open-source repositories to identify unreported security patches. The sheer volume of commits makes this task infeasible to accomplish manually. Consequently, security patch detectors commonly trained and evaluated on security patches linked from vulnerability reports in the National Vulnerability Database (NVD). In this study, we assess the effectiveness of these detectors when applied in-the-wild. Our results show that models trained on NVD-derived data show substantially decreased performance, with decreases in F1-score of up to 90\% when tested on in-the-wild security patches, rendering them impractical for real-world use. An analysis comparing security patches identified in-the-wild and commits linked from NVD reveals that they can be easily distinguished from each other. Security patches associated with NVD have different distribution of commit messages, vulnerability types, and composition of changes. These differences suggest that NVD may be unsuitable as the \textit{sole} source of data for training models to detect security patches. We find that constructing a dataset that combines security patches from NVD data with a small subset of manually identified security patches can improve model robustness.

Related papers

• Vulnerability Localization Benchmark: Measuring Agentic Security Analysis at Repository Scale [54.83971397981572]
  Language-model agents increasingly operate over complete software repositories.<n> cybersecurity evaluations measure whether they can detect, reproduce, or repair vulnerabilities rather than whether they can locate the relevant code.<n>We study vulnerability localization: given a weakness class and an unfamiliar repository, identify the implementation files associated with that weakness.
  arXiv  Detail & Related papers  (2026-09-14T17:44:51Z)
• PatchBench: Evaluating AI Agents for Vulnerability Patching [9.083611812336539]
  We propose a new benchmark for evaluating AI agents on realistic vulnerability patching tasks.<n> PatchBench selects vulnerabilities whose ground-truth fixes lie outside the crash stack and uses vulnerability transplant and code mutations to migrate historical vulnerabilities into new repository contexts.<n>Across 11 state-of-the-art agents, including the top three AIxCC agents, the original PoC-only validations the patching task solve rate of agents by 1.83$times$ on average.
  arXiv  Detail & Related papers  (2026-09-03T16:44:22Z)
• SecPI: Secure Code Generation with Reasoning Models via Security Reasoning Internalization [50.71047638695205]
  Reasoning language models (RLMs) are increasingly used in programming.<n>Yet, even state-of-the-art RLMs frequently introduce critical security vulnerabilities in generated code.<n>We present SecPI, a fine-tuning pipeline that teaches RLMs to internalize structured security reasoning.
  arXiv  Detail & Related papers  (2026-04-04T04:29:11Z)
• DiffuGuard: How Intrinsic Safety is Lost and Found in Diffusion Large Language Models [50.21378052667732]
  We conduct an in-depth analysis of dLLM vulnerabilities to jailbreak attacks across two distinct dimensions: intra-step and inter-step dynamics.<n>We propose DiffuGuard, a training-free defense framework that addresses vulnerabilities through a dual-stage approach.
  arXiv  Detail & Related papers  (2025-09-29T05:17:10Z)
• Weakly Supervised Vulnerability Localization via Multiple Instance Learning [46.980136742826836]
  We propose a novel approach called WAVES for WeAkly supervised Vulnerability localization via multiplE inStance learning.<n>WAVES has the capability to determine whether a function is vulnerable (i.e., vulnerability detection) and pinpoint the vulnerable statements.<n>Our approach achieves comparable performance in vulnerability detection and state-of-the-art performance in statement-level vulnerability localization.
  arXiv  Detail & Related papers  (2025-09-14T15:11:39Z)
• Defending against Indirect Prompt Injection by Instruction Detection [109.30156975159561]
  InstructDetector is a novel detection-based approach that leverages the behavioral states of LLMs to identify potential IPI attacks.<n>InstructDetector achieves a detection accuracy of 99.60% in the in-domain setting and 96.90% in the out-of-domain setting, and reduces the attack success rate to just 0.03% on the BIPIA benchmark.
  arXiv  Detail & Related papers  (2025-05-08T13:04:45Z)
• Fast and Accurate Silent Vulnerability Fix Retrieval [7.512949497610182]
  Existing approaches to trace/retrieve the patching commit for fixing a CVE suffer from two major challenges.<n>We propose SITPatchTracer, a scalable and effective retrieval system for tracing known vulnerability patches.<n>Using SITPatchTracer, we have successfully traced and merged the patch links for 35 new CVEs in the GitHub Advisory database.
  arXiv  Detail & Related papers  (2025-03-29T01:53:07Z)
• CommitShield: Tracking Vulnerability Introduction and Fix in Version Control Systems [15.037460085046806]
  CommitShield is a tool for detecting vulnerabilities in code commits.<n>It combines the code analysis capabilities of static analysis tools with the natural language and code understanding capabilities of large language models.<n>We show that CommitShield improves recall by 76%-87% over state-of-the-art methods in the vulnerability fix detection task.
  arXiv  Detail & Related papers  (2025-01-07T08:52:55Z)
• Detecting and Interpreting NSFW Prompts in Text-to-Image Models through Uncovering Harmful Semantics [48.93115678681624]
  Malicious users exploit text-to-image (T2I) models to generate Not-Safe-for-Work (NSFW) images.<n>We propose HiddenGuard, a framework leveraging the hidden states of T2I models to detect NSFW prompts.<n>Our experiments show that HiddenGuard significantly outperforms both commercial and open-source moderation tools.
  arXiv  Detail & Related papers  (2024-12-24T03:17:45Z)
• Fixing Security Vulnerabilities with AI in OSS-Fuzz [9.730566646484304]
  OSS-Fuzz is the most significant and widely used infrastructure for continuous validation of open source systems. We customise the well-known AutoCodeRover agent for fixing security vulnerabilities. Our experience with OSS-Fuzz vulnerability data shows that LLM agent autonomy is useful for successful security patching.
  arXiv  Detail & Related papers  (2024-11-03T16:20:32Z)
• VERCATION: Precise Vulnerable Open-source Software Version Identification based on Static Analysis and LLM [12.706661324384319]
  Open-source software (OSS) has experienced a surge in popularity, attributed to its collaborative development model and cost-effective nature.<n>The adoption of specific software versions in development projects may introduce security risks when these versions bring along vulnerabilities.<n>This paper presents VERCATION, an approach designed to identify vulnerable versions of OSS written in C/C++.
  arXiv  Detail & Related papers  (2024-08-14T06:43:06Z)
• Watch the Watcher! Backdoor Attacks on Security-Enhancing Diffusion Models [65.30406788716104]
  This work investigates the vulnerabilities of security-enhancing diffusion models. We demonstrate that these models are highly susceptible to DIFF2, a simple yet effective backdoor attack. Case studies show that DIFF2 can significantly reduce both post-purification and certified accuracy across benchmark datasets and models.
  arXiv  Detail & Related papers  (2024-06-14T02:39:43Z)
• Just-in-Time Detection of Silent Security Patches [7.840762542485285]
  Security patches can be em silent, i.e., they do not always come with comprehensive advisories such as CVEs. This lack of transparency leaves users oblivious to available security updates, providing ample opportunity for attackers to exploit unpatched vulnerabilities. We propose to leverage large language models (LLMs) to augment patch information with generated code change explanations.
  arXiv  Detail & Related papers  (2023-12-02T22:53:26Z)
• VELVET: a noVel Ensemble Learning approach to automatically locate VulnErable sTatements [62.93814803258067]
  This paper presents VELVET, a novel ensemble learning approach to locate vulnerable statements in source code. Our model combines graph-based and sequence-based neural networks to successfully capture the local and global context of a program graph. VELVET achieves 99.6% and 43.6% top-1 accuracy over synthetic data and real-world data, respectively.
  arXiv  Detail & Related papers  (2021-12-20T22:45:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.