A Ground-Truth-Based Evaluation of Vulnerability Detection Across Multiple Ecosystems

• URL: http://arxiv.org/abs/2604.21111v1
• Date: Wed, 22 Apr 2026 21:52:58 GMT
• Title: A Ground-Truth-Based Evaluation of Vulnerability Detection Across Multiple Ecosystems
• Abstract summary: We present an empirical evaluation of vulnerability detection across multiple software ecosystems using a curated ground-truth dataset from the Open Source Vulnerabilities (OSV) database.<n>The dataset explicitly maps vulnerabilities to concrete package versions and enables a systematic comparison of detection results across different tools and services.
• Score: 0.1057621049174714
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Automated vulnerability detection tools are widely used to identify security vulnerabilities in software dependencies. However, the evaluation of such tools remains challenging due to the heterogeneous structure of vulnerability data sources, inconsistent identifier schemes, and ambiguities in version range specifications. In this paper, we present an empirical evaluation of vulnerability detection across multiple software ecosystems using a curated ground-truth dataset derived from the Open Source Vulnerabilities (OSV) database. The dataset explicitly maps vulnerabilities to concrete package versions and enables a systematic comparison of detection results across different tools and services. Since vulnerability databases such as OSV are continuously updated, the dataset used in this study represents a snapshot of the vulnerability landscape at the time of the evaluation. To support reproducibility and future studies, we provide an open-source tool that automatically reconstructs the dataset from the current OSV database using the methodology described in this paper. Our evaluation highlights systematic differences between vulnerability detection systems and demonstrates the importance of transparent dataset construction for reproducible empirical security research.

Related papers

• Vulnerability Localization Benchmark: Measuring Agentic Security Analysis at Repository Scale [54.83971397981572]
  Language-model agents increasingly operate over complete software repositories.<n> cybersecurity evaluations measure whether they can detect, reproduce, or repair vulnerabilities rather than whether they can locate the relevant code.<n>We study vulnerability localization: given a weakness class and an unfamiliar repository, identify the implementation files associated with that weakness.
  arXiv  Detail & Related papers  (2026-09-14T17:44:51Z)
• The Distributed Open-Source Vulnerability Ecosystem [0.06423291545613526]
  This paper describes vulnerability management as a distributed process of information exchange and transformation.<n>It identifies heterogeneous information sources, divergent identity and version models, temporal change, and context-dependent assessment as major causes of inconsistent scanner findings.
  arXiv  Detail & Related papers  (2026-07-16T12:17:28Z)
• ARVO: Atlas of Reproducible Vulnerabilities for Open-Source Software [19.754448461368323]
  We produce a new security dataset which ensures for diverse vulnerabilities at scale.<n>In our evaluation, ARVO successfully reproduces 81% of vulnerabilities and achieves 89.4% accuracy on the located patches.
  arXiv  Detail & Related papers  (2026-06-15T20:49:29Z)
• Conflicting Scores, Confusing Signals: An Empirical Study of Vulnerability Scoring Systems [4.286337370812793]
  This work provides the first large-scale, outcome-linked empirical comparison of four publicly available vulnerability scoring systems.<n>We use a dataset of 600 real-world vulnerabilities derived from four months of Microsoft's Patch Tuesday disclosures.
  arXiv  Detail & Related papers  (2025-08-19T08:54:58Z)
• Deep Learning Models for Robust Facial Liveness Detection [56.08694048252482]
  This study introduces a robust solution through novel deep learning models addressing the deficiencies in contemporary anti-spoofing techniques.<n>By innovatively integrating texture analysis and reflective properties associated with genuine human traits, our models distinguish authentic presence from replicas with remarkable precision.
  arXiv  Detail & Related papers  (2025-08-12T17:19:20Z)
• DATABench: Evaluating Dataset Auditing in Deep Learning from an Adversarial Perspective [70.77570343385928]
  We introduce a novel taxonomy, classifying existing methods based on their reliance on internal features (IF) (inherent to the data) versus external features (EF) (artificially introduced for auditing)<n>We formulate two primary attack types: evasion attacks, designed to conceal the use of a dataset, and forgery attacks, intending to falsely implicate an unused dataset.<n>Building on the understanding of existing methods and attack objectives, we further propose systematic attack strategies: decoupling, removal, and detection for evasion; adversarial example-based methods for forgery.<n>Our benchmark, DATABench, comprises 17 evasion attacks, 5 forgery attacks, and 9
  arXiv  Detail & Related papers  (2025-07-08T03:07:15Z)
• VulStamp: Vulnerability Assessment using Large Language Model [28.25412570467278]
  VulStamp is a novel intention-guided framework to facilitate description-free vulnerability assessment.<n>Based on the intention information, VulStamp uses a prompt-tuned model for vulnerability assessment.
  arXiv  Detail & Related papers  (2025-06-13T06:14:56Z)
• VulZoo: A Comprehensive Vulnerability Intelligence Dataset [12.229092589037808]
  VulZoo is a comprehensive vulnerability intelligence dataset that covers 17 popular vulnerability information sources. We make VulZoo publicly available and maintain it with incremental updates to facilitate future research.
  arXiv  Detail & Related papers  (2024-06-24T06:39:07Z)
• DeepfakeBench: A Comprehensive Benchmark of Deepfake Detection [55.70982767084996]
  A critical yet frequently overlooked challenge in the field of deepfake detection is the lack of a standardized, unified, comprehensive benchmark. We present the first comprehensive benchmark for deepfake detection, called DeepfakeBench, which offers three key contributions. DeepfakeBench contains 15 state-of-the-art detection methods, 9CL datasets, a series of deepfake detection evaluation protocols and analysis tools, as well as comprehensive evaluations.
  arXiv  Detail & Related papers  (2023-07-04T01:34:41Z)
• VELVET: a noVel Ensemble Learning approach to automatically locate VulnErable sTatements [62.93814803258067]
  This paper presents VELVET, a novel ensemble learning approach to locate vulnerable statements in source code. Our model combines graph-based and sequence-based neural networks to successfully capture the local and global context of a program graph. VELVET achieves 99.6% and 43.6% top-1 accuracy over synthetic data and real-world data, respectively.
  arXiv  Detail & Related papers  (2021-12-20T22:45:27Z)
• Survey of Network Intrusion Detection Methods from the Perspective of the Knowledge Discovery in Databases Process [63.75363908696257]
  We review the methods that have been applied to network data with the purpose of developing an intrusion detector. We discuss the techniques used for the capture, preparation and transformation of the data, as well as, the data mining and evaluation methods. As a result of this literature review, we investigate some open issues which will need to be considered for further research in the area of network security.
  arXiv  Detail & Related papers  (2020-01-27T11:21:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.