An Empirical Analysis of Vulnerability Detection Tools for Solidity Smart Contracts Using Line Level Manually Annotated Vulnerabilities

• URL: http://arxiv.org/abs/2505.15756v1
• Date: Wed, 21 May 2025 17:01:18 GMT
• Title: An Empirical Analysis of Vulnerability Detection Tools for Solidity Smart Contracts Using Line Level Manually Annotated Vulnerabilities
• Authors: Francesco Salzano, Cosmo Kevin Antenucci, Simone Scalabrino, Giovanni Rosa, Rocco Oliveto, Remo Pareschi,
• Abstract summary: This paper provides an empirical evaluation of automated vulnerability analysis tools specifically designed for Solidity smart contracts.<n>We conducted an assessment using an annotated dataset of 2,182 instances we manually annotated with line-level vulnerability labels.<n>We identified a set of 3 tools that, combined, achieve up to 76.78% found vulnerabilities taking less than one minute to run.
• Score: 5.357551358237259
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The rapid adoption of blockchain technology highlighted the importance of ensuring the security of smart contracts due to their critical role in automated business logic execution on blockchain platforms. This paper provides an empirical evaluation of automated vulnerability analysis tools specifically designed for Solidity smart contracts. Leveraging the extensive SmartBugs 2.0 framework, which includes 20 analysis tools, we conducted a comprehensive assessment using an annotated dataset of 2,182 instances we manually annotated with line-level vulnerability labels. Our evaluation highlights the detection effectiveness of these tools in detecting various types of vulnerabilities, as categorized by the DASP TOP 10 taxonomy. We evaluated the effectiveness of a Large Language Model-based detection method on two popular datasets. In this case, we obtained inconsistent results with the two datasets, showing unreliable detection when analyzing real-world smart contracts. Our study identifies significant variations in the accuracy and reliability of different tools and demonstrates the advantages of combining multiple detection methods to improve vulnerability identification. We identified a set of 3 tools that, combined, achieve up to 76.78\% found vulnerabilities taking less than one minute to run, on average. This study contributes to the field by releasing the largest dataset of manually analyzed smart contracts with line-level vulnerability annotations and the empirical evaluation of the greatest number of tools to date.

Related papers

• DATABench: Evaluating Dataset Auditing in Deep Learning from an Adversarial Perspective [59.66984417026933]
  We introduce a novel taxonomy, classifying existing methods based on their reliance on internal features (IF) (inherent to the data) versus external features (EF) (artificially introduced for auditing)<n>We formulate two primary attack types: evasion attacks, designed to conceal the use of a dataset, and forgery attacks, intending to falsely implicate an unused dataset.<n>Building on the understanding of existing methods and attack objectives, we further propose systematic attack strategies: decoupling, removal, and detection for evasion; adversarial example-based methods for forgery.<n>Our benchmark, DATABench, comprises 17 evasion attacks, 5 forgery attacks, and 9
  arXiv  Detail & Related papers  (2025-07-08T03:07:15Z)
• FORGE: An LLM-driven Framework for Large-Scale Smart Contract Vulnerability Dataset Construction [34.20628333535654]
  FORGE is the first automated approach for constructing smart contract vulnerability datasets.<n>We generate a dataset comprising 81,390 solidity files and 27,497 vulnerability findings across 296 CWE categories.<n>Results reveal the significant limitations in current detection capabilities.
  arXiv  Detail & Related papers  (2025-06-23T16:03:16Z)
• Automated Vulnerability Injection in Solidity Smart Contracts: A Mutation-Based Approach for Benchmark Development [2.0074256613821033]
  This work evaluates whether mutation seeding can effectively inject vulnerabilities into Solidity-based smart contracts.<n>We propose MuSe, a tool to generate vulnerable smart contracts by leveraging pattern-based mutation operators.<n>We analyzed these vulnerable smart contracts using Slither, a static analysis tool, to determine its capacity to identify them and assess their validity.
  arXiv  Detail & Related papers  (2025-04-22T14:46:18Z)
• Impact of Code Transformation on Detection of Smart Contract Vulnerabilities [0.0]
  This paper presents a method for improving the quantity and quality of smart contract vulnerability datasets. The approach centers around semantic-preserving code transformation, a technique that modifies the source code structure without altering its semantic meaning. The improved results show that many newly created vulnerabilities can bypass tools and the false reporting rate goes up to 100%.
  arXiv  Detail & Related papers  (2024-10-29T03:08:25Z)
• Static Application Security Testing (SAST) Tools for Smart Contracts: How Far Are We? [14.974832502863526]
  In recent years, the importance of smart contract security has been heightened by the increasing number of attacks against them. To address this issue, a multitude of static application security testing (SAST) tools have been proposed for detecting vulnerabilities in smart contracts. In this paper, we propose an up-to-date and fine-grained taxonomy that includes 45 unique vulnerability types for smart contracts.
  arXiv  Detail & Related papers  (2024-04-28T13:40:18Z)
• ASSERT: Automated Safety Scenario Red Teaming for Evaluating the Robustness of Large Language Models [65.79770974145983]
  ASSERT, Automated Safety Scenario Red Teaming, consists of three methods -- semantically aligned augmentation, target bootstrapping, and adversarial knowledge injection. We partition our prompts into four safety domains for a fine-grained analysis of how the domain affects model performance. We find statistically significant performance differences of up to 11% in absolute classification accuracy among semantically related scenarios and error rates of up to 19% absolute error in zero-shot adversarial settings.
  arXiv  Detail & Related papers  (2023-10-14T17:10:28Z)
• Identifying Vulnerabilities in Smart Contracts using Interval Analysis [0.0]
  This paper focuses on utilizing interval analysis, an existing static analysis method, for detecting vulnerabilities in smart contracts. We present a selection of motivating examples featuring vulnerable smart contracts and share the results from our experiments conducted with various existing detection tools.
  arXiv  Detail & Related papers  (2023-09-25T01:17:56Z)
• Enhancing Smart Contract Security Analysis with Execution Property Graphs [48.31617821205042]
  We introduce Clue, a dynamic analysis framework specifically designed for a runtime virtual machine.<n>Clue captures critical information during contract executions, employing a novel graph-based representation, the Execution Property Graph.<n> evaluation results reveal Clue's superior performance with high true positive rates and low false positive rates, outperforming state-of-the-art tools.
  arXiv  Detail & Related papers  (2023-05-23T13:16:42Z)
• Improving robustness of jet tagging algorithms with adversarial training [56.79800815519762]
  We investigate the vulnerability of flavor tagging algorithms via application of adversarial attacks. We present an adversarial training strategy that mitigates the impact of such simulated attacks.
  arXiv  Detail & Related papers  (2022-03-25T19:57:19Z)
• VELVET: a noVel Ensemble Learning approach to automatically locate VulnErable sTatements [62.93814803258067]
  This paper presents VELVET, a novel ensemble learning approach to locate vulnerable statements in source code. Our model combines graph-based and sequence-based neural networks to successfully capture the local and global context of a program graph. VELVET achieves 99.6% and 43.6% top-1 accuracy over synthetic data and real-world data, respectively.
  arXiv  Detail & Related papers  (2021-12-20T22:45:27Z)
• Software Vulnerability Detection via Deep Learning over Disaggregated Code Graph Representation [57.92972327649165]
  This work explores a deep learning approach to automatically learn the insecure patterns from code corpora. Because code naturally admits graph structures with parsing, we develop a novel graph neural network (GNN) to exploit both the semantic context and structural regularity of a program.
  arXiv  Detail & Related papers  (2021-09-07T21:24:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.