SecureFalcon: Are We There Yet in Automated Software Vulnerability Detection with LLMs?

• URL: http://arxiv.org/abs/2307.06616v2
• Date: Wed, 29 May 2024 18:22:48 GMT
• Title: SecureFalcon: Are We There Yet in Automated Software Vulnerability Detection with LLMs?
• Authors: Mohamed Amine Ferrag, Ammar Battah, Norbert Tihanyi, Ridhi Jain, Diana Maimut, Fatima Alwahedi, Thierry Lestable, Narinderjit Singh Thandi, Abdechakour Mechri, Merouane Debbah, Lucas C. Cordeiro,
• Abstract summary: We introduce SecureFalcon, an innovative model architecture with only 121 million parameters derived from the Falcon-40B model. SecureFalcon achieves 94% accuracy in binary classification and up to 92% in multiclassification, with instant CPU inference times.
• Score: 3.566250952750758
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Software vulnerabilities can cause numerous problems, including crashes, data loss, and security breaches. These issues greatly compromise quality and can negatively impact the market adoption of software applications and systems. Traditional bug-fixing methods, such as static analysis, often produce false positives. While bounded model checking, a form of Formal Verification (FV), can provide more accurate outcomes compared to static analyzers, it demands substantial resources and significantly hinders developer productivity. Can Machine Learning (ML) achieve accuracy comparable to FV methods and be used in popular instant code completion frameworks in near real-time? In this paper, we introduce SecureFalcon, an innovative model architecture with only 121 million parameters derived from the Falcon-40B model and explicitly tailored for classifying software vulnerabilities. To achieve the best performance, we trained our model using two datasets, namely the FormAI dataset and the FalconVulnDB. The FalconVulnDB is a combination of recent public datasets, namely the SySeVR framework, Draper VDISC, Bigvul, Diversevul, SARD Juliet, and ReVeal datasets. These datasets contain the top 25 most dangerous software weaknesses, such as CWE-119, CWE-120, CWE-476, CWE-122, CWE-190, CWE-121, CWE-78, CWE-787, CWE-20, and CWE-762. SecureFalcon achieves 94% accuracy in binary classification and up to 92% in multiclassification, with instant CPU inference times. It outperforms existing models such as BERT, RoBERTa, CodeBERT, and traditional ML algorithms, promising to push the boundaries of software vulnerability detection and instant code completion frameworks.

Related papers

• CyberLLMInstruct: A New Dataset for Analysing Safety of Fine-Tuned LLMs Using Cyber Security Data [2.2530496464901106]
  integration of large language models into cyber security applications presents significant opportunities. CyberLLMInstruct is a dataset of 54,928 instruction-response pairs spanning cyber security tasks. Fine-tuning models can achieve up to 92.50 percent accuracy on the CyberMetric benchmark.
  arXiv  Detail & Related papers  (2025-03-12T12:29:27Z)
• Enhanced LLM-Based Framework for Predicting Null Pointer Dereference in Source Code [2.2020053359163305]
  We propose a novel approach using a fine-tuned Large Language Model (LLM) termed "DeLLNeuN" Our model showed 87% accuracy with 88% precision using the Draper VDISC dataset.
  arXiv  Detail & Related papers  (2024-11-29T19:24:08Z)
• CleanVul: Automatic Function-Level Vulnerability Detection in Code Commits Using LLM Heuristics [12.053158610054911]
  This paper introduces the first methodology that uses the Large Language Model (LLM) with a enhancement to automatically identify vulnerability-fixing changes from VFCs. VulSifter was applied to a large-scale study, where we conducted a crawl of 127,063 repositories on GitHub, resulting in the acquisition of 5,352,105 commits. We then developed CleanVul, a high-quality dataset comprising 11,632 functions using our LLM enhancement approach.
  arXiv  Detail & Related papers  (2024-11-26T09:51:55Z)
• FaultGuard: A Generative Approach to Resilient Fault Prediction in Smart Electrical Grids [53.2306792009435]
  FaultGuard is the first framework for fault type and zone classification resilient to adversarial attacks. We propose a low-complexity fault prediction model and an online adversarial training technique to enhance robustness. Our model outclasses the state-of-the-art for resilient fault prediction benchmarking, with an accuracy of up to 0.958.
  arXiv  Detail & Related papers  (2024-03-26T08:51:23Z)
• Unveiling Hidden Links Between Unseen Security Entities [3.7138962865789353]
  VulnScopper is an innovative approach that utilizes multi-modal representation learning, combining Knowledge Graphs (KG) and Natural Processing (NLP) We evaluate VulnScopper on two major security datasets, the National Vulnerability Database (NVD) and the Red Hat CVE database. Our results show that VulnScopper outperforms existing methods, achieving up to 78% Hits@10 accuracy in linking CVEs to Common Vulnerabilities and Exposures (CWEs), and Common Platform Languageions (CPEs)
  arXiv  Detail & Related papers  (2024-03-04T13:14:39Z)
• Data-Free Hard-Label Robustness Stealing Attack [67.41281050467889]
  We introduce a novel Data-Free Hard-Label Robustness Stealing (DFHL-RS) attack in this paper. It enables the stealing of both model accuracy and robustness by simply querying hard labels of the target model. Our method achieves a clean accuracy of 77.86% and a robust accuracy of 39.51% against AutoAttack.
  arXiv  Detail & Related papers  (2023-12-10T16:14:02Z)
• Vulnerability Detection Using Two-Stage Deep Learning Models [0.0]
  Two deep learning models were proposed for vulnerability detection in C/C++ source codes. The first stage is CNN which detects if the source code contains any vulnerability. The second stage is CNN-LTSM that classifies this vulnerability into a class of 50 different types of vulnerabilities.
  arXiv  Detail & Related papers  (2023-05-08T22:12:34Z)
• CodeLMSec Benchmark: Systematically Evaluating and Finding Security Vulnerabilities in Black-Box Code Language Models [58.27254444280376]
  Large language models (LLMs) for automatic code generation have achieved breakthroughs in several programming tasks. Training data for these models is usually collected from the Internet (e.g., from open-source repositories) and is likely to contain faults and security vulnerabilities. This unsanitized training data can cause the language models to learn these vulnerabilities and propagate them during the code generation procedure.
  arXiv  Detail & Related papers  (2023-02-08T11:54:07Z)
• Fast and Accurate Error Simulation for CNNs against Soft Errors [64.54260986994163]
  We present a framework for the reliability analysis of Conal Neural Networks (CNNs) via an error simulation engine. These error models are defined based on the corruption patterns of the output of the CNN operators induced by faults. We show that our methodology achieves about 99% accuracy of the fault effects w.r.t. SASSIFI, and a speedup ranging from 44x up to 63x w.r.t.FI, that only implements a limited set of error models.
  arXiv  Detail & Related papers  (2022-06-04T19:45:02Z)
• VELVET: a noVel Ensemble Learning approach to automatically locate VulnErable sTatements [62.93814803258067]
  This paper presents VELVET, a novel ensemble learning approach to locate vulnerable statements in source code. Our model combines graph-based and sequence-based neural networks to successfully capture the local and global context of a program graph. VELVET achieves 99.6% and 43.6% top-1 accuracy over synthetic data and real-world data, respectively.
  arXiv  Detail & Related papers  (2021-12-20T22:45:27Z)
• Federated Learning with Unreliable Clients: Performance Analysis and Mechanism Design [76.29738151117583]
  Federated Learning (FL) has become a promising tool for training effective machine learning models among distributed clients. However, low quality models could be uploaded to the aggregator server by unreliable clients, leading to a degradation or even a collapse of training. We model these unreliable behaviors of clients and propose a defensive mechanism to mitigate such a security risk.
  arXiv  Detail & Related papers  (2021-05-10T08:02:27Z)
• V2W-BERT: A Framework for Effective Hierarchical Multiclass Classification of Software Vulnerabilities [7.906207218788341]
  We present a novel Transformer-based learning framework (V2W-BERT) in this paper. By using ideas from natural language processing, link prediction and transfer learning, our method outperforms previous approaches. We achieve up to 97% prediction accuracy for randomly partitioned data and up to 94% prediction accuracy in temporally partitioned data.
  arXiv  Detail & Related papers  (2021-02-23T05:16:57Z)
• SIGL: Securing Software Installations Through Deep Graph Learning [25.178164770390712]
  Recent supply-chain attacks demonstrate that application integrity must be ensured during installation itself. We introduce SIGL, a new tool for detecting malicious behavior during software installation. We demonstrate that SIGL has a detection accuracy of 96%, outperforming similar systems from industry and academia.
  arXiv  Detail & Related papers  (2020-08-26T12:52:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.