ChatGPT's Potential in Cryptography Misuse Detection: A Comparative Analysis with Static Analysis Tools

• URL: http://arxiv.org/abs/2409.06561v1
• Date: Tue, 10 Sep 2024 14:50:12 GMT
• Title: ChatGPT's Potential in Cryptography Misuse Detection: A Comparative Analysis with Static Analysis Tools
• Authors: Ehsan Firouzi, Mohammad Ghafari, Mike Ebrahimi,
• Abstract summary: cryptography misuse detectors have demonstrated inconsistent performance and remain largely inaccessible to most developers. We investigated the extent to which ChatGPT can detect cryptography misuses and compared its performance with that of the state-of-the-art static analysis tools. Our investigation, mainly based on the CryptoAPI-Bench benchmark, demonstrated that ChatGPT is effective in identifying cryptography API misuses, and with the use of prompt engineering, it can even outperform leading static cryptography misuse detectors.
• Score: 0.08192907805418582
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The correct adoption of cryptography APIs is challenging for mainstream developers, often resulting in widespread API misuse. Meanwhile, cryptography misuse detectors have demonstrated inconsistent performance and remain largely inaccessible to most developers. We investigated the extent to which ChatGPT can detect cryptography misuses and compared its performance with that of the state-of-the-art static analysis tools. Our investigation, mainly based on the CryptoAPI-Bench benchmark, demonstrated that ChatGPT is effective in identifying cryptography API misuses, and with the use of prompt engineering, it can even outperform leading static cryptography misuse detectors.

Related papers

• Anomaly Detection with LWE Encrypted Control [5.263161322684099]
  We present a novel mechanism for anomaly detection over Learning with Errors encrypted signals. The detector exploits the homomorphic property of LWE encryption to perform hypothesis tests on transformations of the encrypted samples.
  arXiv  Detail & Related papers  (2025-02-14T16:38:51Z)
• Cryptanalysis via Machine Learning Based Information Theoretic Metrics [58.96805474751668]
  We propose two novel applications of machine learning (ML) algorithms to perform cryptanalysis on any cryptosystem. These algorithms can be readily applied in an audit setting to evaluate the robustness of a cryptosystem. We show that our classification model correctly identifies the encryption schemes that are not IND-CPA secure, such as DES, RSA, and AES ECB, with high accuracy.
  arXiv  Detail & Related papers  (2025-01-25T04:53:36Z)
• Beyond Static Tools: Evaluating Large Language Models for Cryptographic Misuse Detection [0.30693357740321775]
  GPT 4-o-mini surpasses current state-of-the-art static analysis tools on the CryptoAPI and MASC datasets. This study highlights the comparative strengths and limitations of static analysis versus LLM-driven approaches.
  arXiv  Detail & Related papers  (2024-11-14T19:33:08Z)
• Scalable Defect Detection via Traversal on Code Graph [10.860910384163892]
  We introduce QVoG, a graph-based static analysis platform for detecting defects and vulnerabilities. It employs a compressed CPG representation to maintain a reasonable graph size, thereby enhancing the overall query efficiency. For projects consisting of 1,000,000+ lines of code, QVoG can complete analysis in approximately 15 minutes, as opposed to 19 minutes with CodeQL.
  arXiv  Detail & Related papers  (2024-06-12T11:24:52Z)
• FoC: Figure out the Cryptographic Functions in Stripped Binaries with LLMs [54.27040631527217]
  We propose a novel framework called FoC to Figure out the Cryptographic functions in stripped binaries. We first build a binary large language model (FoC-BinLLM) to summarize the semantics of cryptographic functions in natural language. We then build a binary code similarity model (FoC-Sim) upon the FoC-BinLLM to create change-sensitive representations and use it to retrieve similar implementations of unknown cryptographic functions in a database.
  arXiv  Detail & Related papers  (2024-03-27T09:45:33Z)
• Exploring ChatGPT's Capabilities on Vulnerability Management [56.4403395100589]
  We explore ChatGPT's capabilities on 6 tasks involving the complete vulnerability management process with a large-scale dataset containing 70,346 samples. One notable example is ChatGPT's proficiency in tasks like generating titles for software bug reports. Our findings reveal the difficulties encountered by ChatGPT and shed light on promising future directions.
  arXiv  Detail & Related papers  (2023-11-11T11:01:13Z)
• Using ChatGPT as a Static Application Security Testing Tool [0.0]
  ChatGPT has caught a huge amount of attention with its remarkable performance. We study the feasibility of using ChatGPT for vulnerability detection in Python source code.
  arXiv  Detail & Related papers  (2023-08-28T09:21:37Z)
• MASC: A Tool for Mutation-Based Evaluation of Static Crypto-API Misuse Detectors [16.62222783321419]
  This demo paper presents the technical details and usage scenarios of our tool, namely Mutation Analysis for evaluating Static Crypto-API misuse detectors (MASC) We developed $12$ generalizable, usage based mutation operators and three mutation scopes, namely Main Scope, Similarity Scope, and Exhaustive Scope, which can be used to expressively instantiate compilable variants of the crypto-API misuse cases. MASC comes with both Command Line Interface and Web-based front-end, making it practical for users of different levels of expertise.
  arXiv  Detail & Related papers  (2023-08-04T13:22:22Z)
• Deep Learning for Hate Speech Detection: A Comparative Study [54.42226495344908]
  We present here a large-scale empirical comparison of deep and shallow hate-speech detection methods. Our goal is to illuminate progress in the area, and identify strengths and weaknesses in the current state-of-the-art. In doing so we aim to provide guidance as to the use of hate-speech detection in practice, quantify the state-of-the-art, and identify future research directions.
  arXiv  Detail & Related papers  (2022-02-19T03:48:20Z)
• Software Vulnerability Detection via Deep Learning over Disaggregated Code Graph Representation [57.92972327649165]
  This work explores a deep learning approach to automatically learn the insecure patterns from code corpora. Because code naturally admits graph structures with parsing, we develop a novel graph neural network (GNN) to exploit both the semantic context and structural regularity of a program.
  arXiv  Detail & Related papers  (2021-09-07T21:24:36Z)
• Spotting adversarial samples for speaker verification by neural vocoders [102.1486475058963]
  We adopt neural vocoders to spot adversarial samples for automatic speaker verification (ASV) We find that the difference between the ASV scores for the original and re-synthesize audio is a good indicator for discrimination between genuine and adversarial samples. Our codes will be made open-source for future works to do comparison.
  arXiv  Detail & Related papers  (2021-07-01T08:58:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.