An Empirical Security Evaluation of LLM-Generated Cryptographic Rust Code

• URL: http://arxiv.org/abs/2604.27001v1
• Date: Wed, 29 Apr 2026 03:58:55 GMT
• Title: An Empirical Security Evaluation of LLM-Generated Cryptographic Rust Code
• Abstract summary: This study presents an empirical evaluation of cryptographic security in 240 Rust code samples.<n>Only 23.3% of the generated code samples were successfully compiled.<n>All three models exhibit systematic failures, including nonce reuse and API hallucinations.
• Score: 3.345437353879255
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Developers and organizations are using Large Language Models (LLMs) to generate security-critical code more frequently than ever, including cryptographic solutions for their products. This study presents an empirical evaluation of cryptographic security in 240 Rust code samples for two crypto algorithms (AES-256-GCM and ChaCha20-Poly1305) generated by three LLMs (Gemini 2.5 Pro, GPT-4o, and DeepSeek Coder) using four different prompt strategies. For each successfully compiled code sample, CodeQL static analysis and our rule-based crypto-specific analyzer were used to detect vulnerabilities, which are also associated with Common Weakness Enumeration (CWE). The evaluation results revealed that only 23.3% of the generated code samples were successfully compiled. Among the compiled code, CodeQL produced only two false positives, while our rule-based crypto-specific analyzer identified vulnerabilities in 57% of the compiled samples with zero false positives. This demonstrates that general-purpose analysis tools are insufficient for code validation for the experimented crypto algorithms. The compilation success of the two algorithms varied significantly (AES-256-GCM 34.2% versus ChaCha20-Poly1305 12.5%), showing a gap in code generation capabilities. While model choice had no significant effect on compilation success, prompt strategy significantly influenced outcomes (P = 0.002), with chain-of-thought prompting performing 5 times worse than zero-shot. All three models exhibit systematic failures, including nonce reuse and API hallucinations.

Related papers

• Tool-Guided Retrieval-Augmented Repair for Securing LLM-Generated C Code [1.3682073278404534]
  Large language models can generate C code from natural-language descriptions, but resulting programs often contain security vulnerabilities and compilation errors.<n>This work investigates how feedback and retrieval improve reliability of LLM-generated C code.
  arXiv  Detail & Related papers  (2026-07-21T19:01:59Z)
• CryptoScope: Utilizing Large Language Models for Automated Cryptographic Logic Vulnerability Detection [10.620899430085576]
  We introduce CryptoScope, a novel framework for automated cryptographic vulnerability detection powered by Large Language Models (LLMs)<n> CryptoScope combines Chain-of-Thought (CoT) prompting with Retrieval-Augmented Generation (RAG)<n>We evaluate CryptoScope on LLM-CLVA, a benchmark of 92 cases primarily derived from real-world CVE vulnerabilities, complemented by cryptographic challenges.
  arXiv  Detail & Related papers  (2025-08-15T17:07:54Z)
• VulCoCo: A Simple Yet Effective Method for Detecting Vulnerable Code Clones [11.650715913321076]
  VulCoCo is a lightweight and scalable approach to detect vulnerable code clones.<n>We first construct a synthetic benchmark that spans various clone types.<n>Our experiments show that VulCoCoCo outperforms prior state-of-the-art methods in terms of Precision@k and mean average precision (MAP)
  arXiv  Detail & Related papers  (2025-07-22T14:54:57Z)
• Decompiling Smart Contracts with a Large Language Model [51.49197239479266]
  Despite Etherscan's 78,047,845 smart contracts deployed on (as of May 26, 2025), a mere 767,520 ( 1%) are open source.<n>This opacity necessitates the automated semantic analysis of on-chain smart contract bytecode.<n>We introduce a pioneering decompilation pipeline that transforms bytecode into human-readable and semantically faithful Solidity code.
  arXiv  Detail & Related papers  (2025-06-24T13:42:59Z)
• HexaCoder: Secure Code Generation via Oracle-Guided Synthetic Training Data [60.75578581719921]
  Large language models (LLMs) have shown great potential for automatic code generation. Recent studies highlight that many LLM-generated code contains serious security vulnerabilities. We introduce HexaCoder, a novel approach to enhance the ability of LLMs to generate secure codes.
  arXiv  Detail & Related papers  (2024-09-10T12:01:43Z)
• SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal [64.9938658716425]
  SORRY-Bench is a proposed benchmark for evaluating large language models' (LLMs) ability to recognize and reject unsafe user requests.<n>First, existing methods often use coarse-grained taxonomy of unsafe topics, and are over-representing some fine-grained topics.<n>Second, linguistic characteristics and formatting of prompts are often overlooked, like different languages, dialects, and more -- which are only implicitly considered in many evaluations.
  arXiv  Detail & Related papers  (2024-06-20T17:56:07Z)
• Uncovering LLM-Generated Code: A Zero-Shot Synthetic Code Detector via Code Rewriting [78.48355455324688]
  We propose a novel zero-shot synthetic code detector based on the similarity between the original code and its LLM-rewritten variants.<n>Our results demonstrate a significant improvement over existing SOTA synthetic content detectors.
  arXiv  Detail & Related papers  (2024-05-25T08:57:28Z)
• How secure is AI-generated Code: A Large-Scale Comparison of Large Language Models [3.4887856546295333]
  This study compares state-of-the-art Large Language Models (LLMs) on their tendency to generate vulnerabilities when writing C programs.<n>At least 62.07% of the generated programs are vulnerable.
  arXiv  Detail & Related papers  (2024-04-29T01:24:14Z)
• FoC: Figure out the Cryptographic Functions in Stripped Binaries with LLMs [51.898805184427545]
  We propose a novel framework called FoC to Figure out the Cryptographic functions in stripped binaries.<n>We first build a binary large language model (FoC-BinLLM) to summarize the semantics of cryptographic functions in natural language.<n>We then build a binary code similarity model (FoC-Sim) upon the FoC-BinLLM to create change-sensitive representations and use it to retrieve similar implementations of unknown cryptographic functions in a database.
  arXiv  Detail & Related papers  (2024-03-27T09:45:33Z)
• Zero-Shot Detection of Machine-Generated Codes [83.0342513054389]
  This work proposes a training-free approach for the detection of LLMs-generated codes. We find that existing training-based or zero-shot text detectors are ineffective in detecting code. Our method exhibits robustness against revision attacks and generalizes well to Java codes.
  arXiv  Detail & Related papers  (2023-10-08T10:08:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.