A systematic literature Review for Transformer-based Software Vulnerability detection

• URL: http://arxiv.org/abs/2604.24822v1
• Date: Mon, 27 Apr 2026 14:26:19 GMT
• Title: A systematic literature Review for Transformer-based Software Vulnerability detection
• Abstract summary: This SLR critically analysed 80 studies published between 2021 and 2025 that utilised transformer models to identify software vulnerabilities.<n>We classify transformer models into encoder, decoder, and combined architectures and analyse both pre-trained and fine-tuned versions utilized on source code, logs, and smart contracts.
• Score: 0.8409304328108455
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Context: Software vulnerabilities pose significant security threats to software systems, especially as software is increasingly used across many areas of daily life, including health, government, and finance. Recently, transformer-based models have demonstrated promising results in automatic software vulnerability identification due to their robust contextual modelling and representation learning capabilities. Objectives: While numerous systematic literature reviews (SLRs) have examined machine learning and deep learning methods for identifying vulnerabilities, a more transformer-centric analysis remains to be explored. This SLR critically analysed 80 studies published between 2021 and 2025 that utilised transformer models to identify software vulnerabilities. Methods: Using Kitchenhams SLR guidelines, we methodically evaluate current research from various perspectives, encompassing study trends, datasets and sources, programming languages, transformer frameworks, detection detail levels, assessment metrics, reference models, types of vulnerabilities, and experimental configurations. Results: We classify transformer models into encoder, decoder, and combined architectures and analyse both pre-trained and fine-tuned versions utilized on source code, logs, and smart contracts. The results emphasise prevailing research trends, frequently utilised benchmarks, and main baselines. It also uncovers crucial technical issues like data imbalance, interpretability, scalability, and generalization across programming languages. Conclusion: By integrating current evidence and recognising unaddressed research areas, this SLR provides a consolidated resource for researchers and professionals seeking to develop more reliable, precise, and interpretable transformer-based vulnerability identification systems.

Related papers

• Out-of-Distribution Detection for Continual Learning: Design Principles and Benchmarking [44.75780122845172]
  Recent years have witnessed significant progress in the development of machine learning models across a wide range of fields.<n>As these models are deployed in ever-changing real-world scenarios, their ability to remain reliable and adaptive over time becomes increasingly important.
  arXiv  Detail & Related papers  (2025-12-16T22:50:01Z)
• Large Language Models for Unit Test Generation: Achievements, Challenges, and the Road Ahead [15.43943391801509]
  Unit testing is an essential yet laborious technique for verifying software.<n>Large Language Models (LLMs) address this limitation by utilizing by leveraging their data-driven knowledge of code semantics and programming patterns.<n>This framework analyzes the literature regarding core generative strategies and a set of enhancement techniques.
  arXiv  Detail & Related papers  (2025-11-26T13:30:11Z)
• A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic System [56.40989626804489]
  This survey provides the first holistic analysis of Large Language Models-powered software engineering.<n>We review over 150 recent papers and propose a taxonomy along two key dimensions: (1) Solutions, categorized into prompt-based, fine-tuning-based, and agent-based paradigms, and (2) Benchmarks, including tasks such as code generation, translation, and repair.
  arXiv  Detail & Related papers  (2025-10-10T06:56:50Z)
• Revisiting Vulnerability Patch Localization: An Empirical Study and LLM-Based Solution [44.388332647211776]
  Open-source software vulnerability patch detection is a critical component for maintaining software security and ensuring software supply chain integrity.<n>Traditional detection methods face significant scalability challenges when processing large volumes of commit histories.<n>We propose a novel two-stage framework that combines version-driven candidate filtering with large language model-based multi-round dialogue voting.
  arXiv  Detail & Related papers  (2025-09-19T09:09:55Z)
• Veritas: Generalizable Deepfake Detection via Pattern-Aware Reasoning [45.99344620383706]
  We introduce HydraFake, a dataset that simulates real-world challenges with hierarchical generalization testing.<n>Specifically, HydraFake involves diversified deepfake techniques and in-the-wild forgeries, along with rigorous training and evaluation protocol.<n>We propose Veritas, a multi-modal large language model (MLLM) based deepfake detector.
  arXiv  Detail & Related papers  (2025-08-28T17:53:05Z)
• AI Agentic Vulnerability Injection And Transformation with Optimized Reasoning [2.918225266151982]
  We present AVIATOR, the first AI-agentic vulnerability injection workflow.<n>It automatically injects realistic, category-specific vulnerabilities for high-fidelity, diverse, large-scale vulnerability dataset generation.<n>It combines semantic analysis, injection synthesis enhanced with LoRA-based fine-tuning and Retrieval-Augmented Generation, as well as post-injection validation via static analysis and LLM-based discriminators.
  arXiv  Detail & Related papers  (2025-08-28T14:59:39Z)
• A Systematic Literature Review on Detecting Software Vulnerabilities with Large Language Models [2.518519330408713]
  Large Language Models (LLMs) in software engineering have sparked interest in their use for software vulnerability detection.<n>The rapid development of this field has resulted in a fragmented research landscape.<n>This fragmentation makes it difficult to obtain a clear overview of the state-of-the-art or compare and categorize studies meaningfully.
  arXiv  Detail & Related papers  (2025-07-30T13:17:16Z)
• LLMs in Software Security: A Survey of Vulnerability Detection Techniques and Insights [12.424610893030353]
  Large Language Models (LLMs) are emerging as transformative tools for software vulnerability detection.<n>This paper provides a detailed survey of LLMs in vulnerability detection.<n>We address challenges such as cross-language vulnerability detection, multimodal data integration, and repository-level analysis.
  arXiv  Detail & Related papers  (2025-02-10T21:33:38Z)
• Outside the Comfort Zone: Analysing LLM Capabilities in Software Vulnerability Detection [9.652886240532741]
  This paper thoroughly analyses large language models' capabilities in detecting vulnerabilities within source code. We evaluate the performance of six open-source models that are specifically trained for vulnerability detection against six general-purpose LLMs.
  arXiv  Detail & Related papers  (2024-08-29T10:00:57Z)
• SoK: Leveraging Transformers for Malware Analysis [8.999677363643224]
  The introduction of transformers has been an important breakthrough for AI research and application as transformers are the foundation behind Generative AI.<n>A promising application domain for transformers is cybersecurity, in particular the malware domain analysis.<n>This SoK paper aims to provide a comprehensive analysis of transformer-based approaches designed for malware analysis.
  arXiv  Detail & Related papers  (2024-05-27T14:14:07Z)
• VELVET: a noVel Ensemble Learning approach to automatically locate VulnErable sTatements [62.93814803258067]
  This paper presents VELVET, a novel ensemble learning approach to locate vulnerable statements in source code. Our model combines graph-based and sequence-based neural networks to successfully capture the local and global context of a program graph. VELVET achieves 99.6% and 43.6% top-1 accuracy over synthetic data and real-world data, respectively.
  arXiv  Detail & Related papers  (2021-12-20T22:45:27Z)
• Robust and Transferable Anomaly Detection in Log Data using Pre-Trained Language Models [59.04636530383049]
  Anomalies or failures in large computer systems, such as the cloud, have an impact on a large number of users. We propose a framework for anomaly detection in log data, as a major troubleshooting source of system information.
  arXiv  Detail & Related papers  (2021-02-23T09:17:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.