Data and Context Matter: Towards Generalizing AI-based Software Vulnerability Detection

• URL: http://arxiv.org/abs/2508.16625v2
• Date: Mon, 06 Oct 2025 19:18:35 GMT
• Title: Data and Context Matter: Towards Generalizing AI-based Software Vulnerability Detection
• Authors: Rijha Safdar, Danyail Mateen, Syed Taha Ali, M. Umer Ashfaq, Wajahat Hussain,
• Abstract summary: We introduce VulGate, a high quality state of the art dataset that mitigates the shortcomings of prior datasets.<n>We also introduce and benchmark multiple encoder-only and decoder-only models.<n>Our model achieves textbf6.8% improvement in recall on the benchmark BigVul dataset and outperforms others on unseen projects.
• Score: 0.6905053769416639
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: AI-based solutions demonstrate remarkable results in identifying vulnerabilities in software, but research has consistently found that this performance does not generalize to unseen codebases. In this paper, we specifically investigate the impact of model architecture, parameter configuration, and quality of training data on the ability of these systems to generalize. For this purpose, we introduce VulGate, a high quality state of the art dataset that mitigates the shortcomings of prior datasets, by removing mislabeled and duplicate samples, updating new vulnerabilities, incorporating additional metadata, integrating hard samples, and including dedicated test sets. We undertake a series of experiments to demonstrate that improved dataset diversity and quality substantially enhances vulnerability detection. We also introduce and benchmark multiple encoder-only and decoder-only models. We find that encoder-based models outperform other models in terms of accuracy and generalization. Our model achieves \textbf{6.8\%} improvement in recall on the benchmark BigVul dataset and outperforms others on unseen projects, demonstrating enhanced generalizability. Our results highlight the role of data quality and model selection in the development of robust vulnerability detection systems. Our findings suggest a direction for future systems with high cross-project effectiveness.

Related papers

• A Validation Strategy for Deep Learning Models: Evaluating and Enhancing Robustness [0.8532585403388676]
  We propose a validation approach that extracts "weak robust" samples directly from the training dataset via local analysis.<n>These samples, being the most susceptible to perturbations, serve as an early and sensitive indicator of the model's vulnerabilities.<n>We demonstrate the effectiveness of our approach on models trained with CIFAR-10, CIFAR-100, and ImageNet.
  arXiv  Detail & Related papers  (2025-09-23T16:14:14Z)
• AI Agentic Vulnerability Injection And Transformation with Optimized Reasoning [2.918225266151982]
  This paper introduces a novel framework designed to automatically introduce realistic, category-specific vulnerabilities into secure C/C++s to generate datasets.<n>The proposed approach coordinates multiple AI agents that simulate expert reasoning, along with function agents and traditional code analysis tools.<n>Our experimental study on 116 code samples from three different benchmarks suggests that our approach outperforms other techniques with regard to dataset accuracy.
  arXiv  Detail & Related papers  (2025-08-28T14:59:39Z)
• It Only Gets Worse: Revisiting DL-Based Vulnerability Detectors from a Practical Perspective [14.271145160443462]
  VulTegra compares scratch-trained and pre-trained DL models for vulnerability detection.<n>State-of-the-art (SOTA) detectors still suffer from low consistency, limited real-world capabilities, and scalability challenges.
  arXiv  Detail & Related papers  (2025-07-13T08:02:56Z)
• RoHOI: Robustness Benchmark for Human-Object Interaction Detection [78.18946529195254]
  Human-Object Interaction (HOI) detection is crucial for robot-human assistance, enabling context-aware support.<n>We introduce the first benchmark for HOI detection, evaluating model resilience under diverse challenges.<n>Our benchmark, RoHOI, includes 20 corruption types based on the HICO-DET and V-COCO datasets and a new robustness-focused metric.
  arXiv  Detail & Related papers  (2025-07-12T01:58:04Z)
• White-Basilisk: A Hybrid Model for Code Vulnerability Detection [50.49233187721795]
  We introduce White-Basilisk, a novel approach to vulnerability detection that demonstrates superior performance.<n>White-Basilisk achieves results in vulnerability detection tasks with a parameter count of only 200M.<n>This research establishes new benchmarks in code security and provides empirical evidence that compact, efficiently designed models can outperform larger counterparts in specialized tasks.
  arXiv  Detail & Related papers  (2025-07-11T12:39:25Z)
• Ensemble-Based Deepfake Detection using State-of-the-Art Models with Robust Cross-Dataset Generalisation [0.0]
  Machine learning-based Deepfake detection models have achieved impressive results on benchmark datasets.<n>But their performance often deteriorates significantly when evaluated on out-of-distribution data.<n>In this work, we investigate an ensemble-based approach for improving the generalization of deepfake detection systems.
  arXiv  Detail & Related papers  (2025-07-08T13:54:48Z)
• Towards Robust Universal Information Extraction: Benchmark, Evaluation, and Solution [66.11004226578771]
  Existing robust benchmark datasets have two key limitations.<n>They generate only a limited range of perturbations for a single Information Extraction (IE) task.<n>Considering the powerful generation capabilities of Large Language Models (LLMs), we introduce a new benchmark dataset for Robust UIE, called RUIE-Bench.<n>We show that training with only textbf15% of the data leads to an average textbf7.5% relative performance improvement across three IE tasks.
  arXiv  Detail & Related papers  (2025-03-05T05:39:29Z)
• Outside the Comfort Zone: Analysing LLM Capabilities in Software Vulnerability Detection [9.652886240532741]
  This paper thoroughly analyses large language models' capabilities in detecting vulnerabilities within source code. We evaluate the performance of six open-source models that are specifically trained for vulnerability detection against six general-purpose LLMs.
  arXiv  Detail & Related papers  (2024-08-29T10:00:57Z)
• A Simple Background Augmentation Method for Object Detection with Diffusion Model [53.32935683257045]
  In computer vision, it is well-known that a lack of data diversity will impair model performance. We propose a simple yet effective data augmentation approach by leveraging advancements in generative models. Background augmentation, in particular, significantly improves the models' robustness and generalization capabilities.
  arXiv  Detail & Related papers  (2024-08-01T07:40:00Z)
• Revisiting the Performance of Deep Learning-Based Vulnerability Detection on Realistic Datasets [4.385369356819613]
  This paper introduces Real-Vul, a dataset representing real-world scenarios for evaluating vulnerability detection models. evaluating DeepWukong, LineVul, ReVeal, and IVDetect shows a significant drop in performance, with precision decreasing by up to 95 percentage points and F1 scores by up to 91 points. Overfitting is identified as a key issue, and an augmentation technique is proposed, potentially improving performance by up to 30%.
  arXiv  Detail & Related papers  (2024-07-03T13:34:30Z)
• AI-Driven Frameworks for Enhancing Data Quality in Big Data Ecosystems: Error_Detection, Correction, and Metadata Integration [0.0]
  This thesis proposes a novel set of interconnected frameworks aimed at enhancing big data quality comprehensively. Firstly, we introduce new quality metrics and a weighted scoring system for precise data quality assessment. Thirdly, we present a generic framework for detecting various quality anomalies using AI models.
  arXiv  Detail & Related papers  (2024-05-06T21:36:45Z)
• Robustness and Generalization Performance of Deep Learning Models on Cyber-Physical Systems: A Comparative Study [71.84852429039881]
  Investigation focuses on the models' ability to handle a range of perturbations, such as sensor faults and noise. We test the generalization and transfer learning capabilities of these models by exposing them to out-of-distribution (OOD) samples.
  arXiv  Detail & Related papers  (2023-06-13T12:43:59Z)
• An Outlier Exposure Approach to Improve Visual Anomaly Detection Performance for Mobile Robots [76.36017224414523]
  We consider the problem of building visual anomaly detection systems for mobile robots. Standard anomaly detection models are trained using large datasets composed only of non-anomalous data. We tackle the problem of exploiting these data to improve the performance of a Real-NVP anomaly detection model.
  arXiv  Detail & Related papers  (2022-09-20T15:18:13Z)
• Generative Modeling Helps Weak Supervision (and Vice Versa) [87.62271390571837]
  We propose a model fusing weak supervision and generative adversarial networks. It captures discrete variables in the data alongside the weak supervision derived label estimate. It is the first approach to enable data augmentation through weakly supervised synthetic images and pseudolabels.
  arXiv  Detail & Related papers  (2022-03-22T20:24:21Z)
• Anomaly Detection Based on Selection and Weighting in Latent Space [73.01328671569759]
  We propose a novel selection-and-weighting-based anomaly detection framework called SWAD. Experiments on both benchmark and real-world datasets have shown the effectiveness and superiority of SWAD.
  arXiv  Detail & Related papers  (2021-03-08T10:56:38Z)
• Negative Data Augmentation [127.28042046152954]
  We show that negative data augmentation samples provide information on the support of the data distribution. We introduce a new GAN training objective where we use NDA as an additional source of synthetic data for the discriminator. Empirically, models trained with our method achieve improved conditional/unconditional image generation along with improved anomaly detection capabilities.
  arXiv  Detail & Related papers  (2021-02-09T20:28:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.