Conflicting Scores, Confusing Signals: An Empirical Study of Vulnerability Scoring Systems

• URL: http://arxiv.org/abs/2508.13644v1
• Date: Tue, 19 Aug 2025 08:54:58 GMT
• Title: Conflicting Scores, Confusing Signals: An Empirical Study of Vulnerability Scoring Systems
• Abstract summary: This work provides the first large-scale, outcome-linked empirical comparison of four publicly available vulnerability scoring systems.<n>We use a dataset of 600 real-world vulnerabilities derived from four months of Microsoft's Patch Tuesday disclosures.
• Score: 4.286337370812793
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Accurately assessing software vulnerabilities is essential for effective prioritization and remediation. While various scoring systems exist to support this task, their differing goals, methodologies and outputs often lead to inconsistent prioritization decisions. This work provides the first large-scale, outcome-linked empirical comparison of four publicly available vulnerability scoring systems: the Common Vulnerability Scoring System (CVSS), the Stakeholder-Specific Vulnerability Categorization (SSVC), the Exploit Prediction Scoring System (EPSS), and the Exploitability Index. We use a dataset of 600 real-world vulnerabilities derived from four months of Microsoft's Patch Tuesday disclosures to investigate the relationships between these scores, evaluate how they support vulnerability management task, how these scores categorize vulnerabilities across triage tiers, and assess their ability to capture the real-world exploitation risk. Our findings reveal significant disparities in how scoring systems rank the same vulnerabilities, with implications for organizations relying on these metrics to make data-driven, risk-based decisions. We provide insights into the alignment and divergence of these systems, highlighting the need for more transparent and consistent exploitability, risk, and severity assessments.

Related papers

• Conformal Changepoint Localization and Root Cause Analysis with Corrupted Observations [55.76952683833966]
  This paper proposes weighted CONCH (W-CONCH) and weighted CROC (W-CROC) to reduce confidence set size when data may be corrupted.<n>Experiments on image-based and real-world changepoint and root-cause benchmarks show that uncertainty-based weighting substantially reduces confidence set size.
  arXiv  Detail & Related papers  (2026-07-29T05:16:59Z)
• Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system security [57.35851886874902]
  Agentic AI systems execute complex tasks autonomously, but their multi-step trajectories introduce new failure modes that challenge trustworthiness.<n>This survey provides a focused examination of trustworthy agentic AI through two core dimensions that are critical for high-risk deployments.<n>For each dimension, we clarify key concepts, identify where risks emerge along the agent workflow, and summarize stage-targeted mitigation strategies.
  arXiv  Detail & Related papers  (2026-05-17T10:26:37Z)
• CVEs With a CVSS Score Greater Than or Equal to 9 [0.0]
  This paper provides a thorough analysis of the identification and resolution timelines of critical vulnerabilities.<n>A mixed-methods approach is employed, integrating quantitative data from global vulnerability databases.<n>The findings indicate significant delays in public disclosure and patch deployment.<n>Despite faster disclosure, the remediation gap for critical vulnerabilities remains a systemic risk.
  arXiv  Detail & Related papers  (2026-04-22T16:50:25Z)
• An empirical analysis of zero-day vulnerabilities disclosed by the zero day initiative [0.0]
  This study analyzes the Zero Day Initiative (ZDI) vulnerability disclosures reported between January and April 2024, Cole [2025] comprising a total of 415 vulnerabilities.<n>The primary objectives of this work are to identify trends in zero-day vulnerability disclosures, examine severity distributions across vendors, and investigate which vulnerability characteristics are most indicative of high severity.
  arXiv  Detail & Related papers  (2025-12-16T23:15:19Z)
• Community Notes are Vulnerable to Rater Bias and Manipulation [75.34858521118305]
  We evaluate the Community Notes algorithm using simulated data that models realistic rater and note behaviors.<n>We find that the algorithm suppresses a substantial fraction of genuinely helpful notes and is highly sensitive to rater biases.
  arXiv  Detail & Related papers  (2025-11-04T14:39:34Z)
• DATABench: Evaluating Dataset Auditing in Deep Learning from an Adversarial Perspective [59.66984417026933]
  We introduce a novel taxonomy, classifying existing methods based on their reliance on internal features (IF) (inherent to the data) versus external features (EF) (artificially introduced for auditing)<n>We formulate two primary attack types: evasion attacks, designed to conceal the use of a dataset, and forgery attacks, intending to falsely implicate an unused dataset.<n>Building on the understanding of existing methods and attack objectives, we further propose systematic attack strategies: decoupling, removal, and detection for evasion; adversarial example-based methods for forgery.<n>Our benchmark, DATABench, comprises 17 evasion attacks, 5 forgery attacks, and 9
  arXiv  Detail & Related papers  (2025-07-08T03:07:15Z)
• Vulnerability Assessment Combining CVSS Temporal Metrics and Bayesian Networks [0.0]
  This work presents an innovative approach by incorporating the temporal dimension into vulnerability assessment.<n>The proposed approach dynamically computes the Temporal Score and updates the CVSS Base Score by processing data on exploits and fixes from vulnerability databases.
  arXiv  Detail & Related papers  (2025-06-23T14:53:17Z)
• SeCodePLT: A Unified Platform for Evaluating the Security of Code GenAI [58.29510889419971]
  Existing benchmarks for evaluating the security risks and capabilities of code-generating large language models (LLMs) face several key limitations.<n>We introduce a general and scalable benchmark construction framework that begins with manually validated, high-quality seed examples and expands them via targeted mutations.<n>Applying this framework to Python, C/C++, and Java, we build SeCodePLT, a dataset of more than 5.9k samples spanning 44 CWE-based risk categories and three security capabilities.
  arXiv  Detail & Related papers  (2024-10-14T21:17:22Z)
• A Relevance Model for Threat-Centric Ranking of Cybersecurity Vulnerabilities [0.29998889086656577]
  The relentless process of tracking and remediating vulnerabilities is a top concern for cybersecurity professionals. We provide a framework for vulnerability management specifically focused on mitigating threats using adversary criteria derived from MITRE ATT&CK. Our results show an average 71.5% - 91.3% improvement towards the identification of vulnerabilities likely to be targeted and exploited by cyber threat actors.
  arXiv  Detail & Related papers  (2024-06-09T23:29:12Z)
• SecScore: Enhancing the CVSS Threat Metric Group with Empirical Evidences [0.0]
  One of the most widely used vulnerability scoring systems (CVSS) does not address the increasing likelihood of emerging an exploit code. We present SecScore, an innovative vulnerability severity score that enhances CVSS Threat metric group.
  arXiv  Detail & Related papers  (2024-05-14T12:25:55Z)
• Chain-of-Thought Prompting of Large Language Models for Discovering and Fixing Software Vulnerabilities [21.787125867708962]
  Large language models (LLMs) have demonstrated impressive potential in various domains. In this paper, we explore how to leverage LLMs and chain-of-thought (CoT) prompting to address three key software vulnerability analysis tasks. We show substantial superiority of our CoT-inspired prompting over the baselines.
  arXiv  Detail & Related papers  (2024-02-27T05:48:18Z)
• ASSERT: Automated Safety Scenario Red Teaming for Evaluating the Robustness of Large Language Models [65.79770974145983]
  ASSERT, Automated Safety Scenario Red Teaming, consists of three methods -- semantically aligned augmentation, target bootstrapping, and adversarial knowledge injection. We partition our prompts into four safety domains for a fine-grained analysis of how the domain affects model performance. We find statistically significant performance differences of up to 11% in absolute classification accuracy among semantically related scenarios and error rates of up to 19% absolute error in zero-shot adversarial settings.
  arXiv  Detail & Related papers  (2023-10-14T17:10:28Z)
• Binary Classification with Confidence Difference [100.08818204756093]
  This paper delves into a novel weakly supervised binary classification problem called confidence-difference (ConfDiff) classification. We propose a risk-consistent approach to tackle this problem and show that the estimation error bound the optimal convergence rate. We also introduce a risk correction approach to mitigate overfitting problems, whose consistency and convergence rate are also proven.
  arXiv  Detail & Related papers  (2023-10-09T11:44:50Z)
• OutCenTR: A novel semi-supervised framework for predicting exploits of vulnerabilities in high-dimensional datasets [0.0]
  We make use of outlier detection techniques to predict vulnerabilities that are likely to be exploited. We propose a dimensionality reduction technique, OutCenTR, that enhances the baseline outlier detection models. The results of our experiments show on average a 5-fold improvement of F1 score in comparison with state-of-the-art dimensionality reduction techniques.
  arXiv  Detail & Related papers  (2023-04-03T00:34:41Z)
• Free Lunch for Generating Effective Outlier Supervision [46.37464572099351]
  We propose an ultra-effective method to generate near-realistic outlier supervision. Our proposed textttBayesAug significantly reduces the false positive rate over 12.50% compared with the previous schemes.
  arXiv  Detail & Related papers  (2023-01-17T01:46:45Z)
• Exploring Robustness of Unsupervised Domain Adaptation in Semantic Segmentation [74.05906222376608]
  We propose adversarial self-supervision UDA (or ASSUDA) that maximizes the agreement between clean images and their adversarial examples by a contrastive loss in the output space. This paper is rooted in two observations: (i) the robustness of UDA methods in semantic segmentation remains unexplored, which pose a security concern in this field; and (ii) although commonly used self-supervision (e.g., rotation and jigsaw) benefits image tasks such as classification and recognition, they fail to provide the critical supervision signals that could learn discriminative representation for segmentation tasks.
  arXiv  Detail & Related papers  (2021-05-23T01:50:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.