An Extensive Comparison of Static Application Security Testing Tools

• URL: http://arxiv.org/abs/2403.09219v1
• Date: Thu, 14 Mar 2024 09:37:54 GMT
• Title: An Extensive Comparison of Static Application Security Testing Tools
• Authors: Matteo Esposito, Valentina Falaschi, Davide Falessi,
• Abstract summary: Static Application Security Testing Tools (SASTTs) identify software vulnerabilities to support the security and reliability of software applications. Several studies have suggested that alternative solutions may be more effective than SASTTs due to their tendency to generate false alarms. Our SASTTs evaluation is based on a controlled, though synthetic, Java.
• Score: 1.3927943269211593
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Context: Static Application Security Testing Tools (SASTTs) identify software vulnerabilities to support the security and reliability of software applications. Interestingly, several studies have suggested that alternative solutions may be more effective than SASTTs due to their tendency to generate false alarms, commonly referred to as low Precision. Aim: We aim to comprehensively evaluate SASTTs, setting a reliable benchmark for assessing and finding gaps in vulnerability identification mechanisms based on SASTTs or alternatives. Method: Our SASTTs evaluation is based on a controlled, though synthetic, Java codebase. It involves an assessment of 1.5 million test executions, and it features innovative methodological features such as effort-aware accuracy metrics and method-level analysis. Results: Our findings reveal that SASTTs detect a tiny range of vulnerabilities. In contrast to prevailing wisdom, SASTTs exhibit high Precision while falling short in Recall. Conclusions: The paper suggests that enhancing Recall, alongside expanding the spectrum of detected vulnerability types, should be the primary focus for improving SASTTs or alternative approaches, such as machine learning-based vulnerability identification solutions.

Related papers

• Bringing Order Amidst Chaos: On the Role of Artificial Intelligence in Secure Software Engineering [0.0]
  The ever-evolving technological landscape offers both opportunities and threats, creating a dynamic space where chaos and order compete. Secure software engineering (SSE) must continuously address vulnerabilities that endanger software systems. This thesis seeks to bring order to the chaos in SSE by addressing domain-specific differences that impact AI accuracy.
  arXiv  Detail & Related papers  (2025-01-09T11:38:58Z)
• Improving Discovery of Known Software Vulnerability For Enhanced Cybersecurity [0.0]
  Vulnerability detection relies on standardized identifiers such as Common Platformion (CPE) strings. Non-standardized CPE strings issued by software vendors create a significant challenge. Inconsistent naming conventions, and versioning practices lead to mismatches when querying databases.
  arXiv  Detail & Related papers  (2024-12-21T12:43:52Z)
• A Comparative Analysis of Vulnerability Management Tools: Evaluating Nessus, Acunetix, and Nikto for Risk Based Security Solutions [0.0]
  This paper presents a comparative analysis of three widely used vulnerability management tools: Nessus, Acunetix, and Nikto. Each tool is assessed based on its detection accuracy, risk scoring using the Common Vulnerability Scoring System (CVSS), ease of use, automation and reporting capabilities, performance metrics, and cost effectiveness.
  arXiv  Detail & Related papers  (2024-11-28T13:14:24Z)
• A Comprehensive Study on Static Application Security Testing (SAST) Tools for Android [22.558610938860124]
  VulsTotal is a unified evaluation platform for defining and describing tools' supported vulnerability types. We select 11 free and open-sourced SAST tools from a pool of 97 existing options, adhering to clearly defined criteria. We then unify 67 general/common vulnerability types for Android SAST tools.
  arXiv  Detail & Related papers  (2024-10-28T05:10:22Z)
• The Impact of SBOM Generators on Vulnerability Assessment in Python: A Comparison and a Novel Approach [56.4040698609393]
  Software Bill of Materials (SBOM) has been promoted as a tool to increase transparency and verifiability in software composition. Current SBOM generation tools often suffer from inaccuracies in identifying components and dependencies. We propose PIP-sbom, a novel pip-inspired solution that addresses their shortcomings.
  arXiv  Detail & Related papers  (2024-09-10T10:12:37Z)
• Towards Efficient Verification of Constant-Time Cryptographic Implementations [5.433710892250037]
  Constant-time programming discipline is an effective software-based countermeasure against timing side-channel attacks. We put forward practical verification approaches based on a novel synergy of taint analysis and safety verification of self-composed programs. Our approach is implemented as a cross-platform and fully automated tool CT-Prover.
  arXiv  Detail & Related papers  (2024-02-21T03:39:14Z)
• ASSERT: Automated Safety Scenario Red Teaming for Evaluating the Robustness of Large Language Models [65.79770974145983]
  ASSERT, Automated Safety Scenario Red Teaming, consists of three methods -- semantically aligned augmentation, target bootstrapping, and adversarial knowledge injection. We partition our prompts into four safety domains for a fine-grained analysis of how the domain affects model performance. We find statistically significant performance differences of up to 11% in absolute classification accuracy among semantically related scenarios and error rates of up to 19% absolute error in zero-shot adversarial settings.
  arXiv  Detail & Related papers  (2023-10-14T17:10:28Z)
• Adversarial Attacks and Defense for Non-Parametric Two-Sample Tests [73.32304304788838]
  This paper systematically uncovers the failure mode of non-parametric TSTs through adversarial attacks. To enable TST-agnostic attacks, we propose an ensemble attack framework that jointly minimizes the different types of test criteria. To robustify TSTs, we propose a max-min optimization that iteratively generates adversarial pairs to train the deep kernels.
  arXiv  Detail & Related papers  (2022-02-07T11:18:04Z)
• Differential privacy and robust statistics in high dimensions [49.50869296871643]
  High-dimensional Propose-Test-Release (HPTR) builds upon three crucial components: the exponential mechanism, robust statistics, and the Propose-Test-Release mechanism. We show that HPTR nearly achieves the optimal sample complexity under several scenarios studied in the literature.
  arXiv  Detail & Related papers  (2021-11-12T06:36:40Z)
• Bayesian Optimization with Machine Learning Algorithms Towards Anomaly Detection [66.05992706105224]
  In this paper, an effective anomaly detection framework is proposed utilizing Bayesian Optimization technique. The performance of the considered algorithms is evaluated using the ISCX 2012 dataset. Experimental results show the effectiveness of the proposed framework in term of accuracy rate, precision, low-false alarm rate, and recall.
  arXiv  Detail & Related papers  (2020-08-05T19:29:35Z)
• SAMBA: Safe Model-Based & Active Reinforcement Learning [59.01424351231993]
  SAMBA is a framework for safe reinforcement learning that combines aspects from probabilistic modelling, information theory, and statistics. We evaluate our algorithm on a variety of safe dynamical system benchmarks involving both low and high-dimensional state representations. We provide intuition as to the effectiveness of the framework by a detailed analysis of our active metrics and safety constraints.
  arXiv  Detail & Related papers  (2020-06-12T10:40:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.