CryptanalysisBench: Can LLMs do Cryptanalysis?

• URL: http://arxiv.org/abs/2607.18538v1
• Date: Mon, 20 Jul 2026 22:09:40 GMT
• Title: CryptanalysisBench: Can LLMs do Cryptanalysis?
• Abstract summary: We introduce CryptanalysisBench, 191 tasks across six families of cryptographic primitives.<n>Our benchmark consists of three tiers: (i) primitives with known practical breaks; (ii) primitives with no known practical break, evaluated both at full strength and as scaled-down variants.<n>Five frontier models break 65%-86% of Tier 1 schemes, 6-12 Tier-2 schemes at full strength, and 24-61 across all scaled-down variants.
• Score: 76.79639292270497
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Cryptanalysis - the task of finding attacks against cryptographic schemes - sits at the intersection of mathematical reasoning and cybersecurity, two areas where LLMs have advanced fastest. Cryptanalysis represents both a clean testbed for frontier reasoning (as practical attacks can be automatically verified) and a domain with unusually high stakes, since the primitives under study underpin our digital security. In this paper we ask whether LLMs can do cryptanalysis, and find that the answer is increasingly yes. We introduce CryptanalysisBench, 191 tasks across six families of cryptographic primitives (block ciphers, hash functions, etc.) drawn primarily from four NIST standardization competitions. Our benchmark consists of three tiers: (i) primitives with known practical breaks; (ii) primitives with no known practical break, evaluated both at full strength and as scaled-down variants; and (iii) a challenge set of production primitives at the frontier of cryptanalysis. Five frontier models (Claude Opus 4.8, Sonnet 5, Mythos 5, GPT 5.5, and the open-weights GLM 5.2) break 65%-86% of Tier 1 schemes, 6-12 Tier-2 schemes at full strength, and 24-61 across all scaled-down variants. Beyond deriving known results, models produce novel cryptanalysis, such as a key-recovery attack that exploits a design flaw in the SpoC AEAD and an error in KINDI's published CCA-security proof, both to the best of our knowledge not previously known. We release CryptanalysisBench as a tool to help track if (or when) AI cryptanalysis becomes a serious factor and as a scaffold for stress-testing candidate schemes before deployment. The attacks that the benchmark already surfaces are an early snapshot of a fast-moving frontier that may soon match, and in places exceed, the published state of the art.

Related papers

• CREBench: Evaluating Large Language Models in Cryptographic Binary Reverse Engineering [12.401873262343862]
  We study the cryptographic binary reverse engineering capabilities of large language models (LLMs)<n>Our benchmark comprises 432 challenges built from 48 standard cryptographic algorithms, 3 insecure crypto key usage scenarios, and 3 difficulty levels.<n>We also establish a strong human expert baseline of 92.19 points, showing that humans maintain an advantage in cryptographic RE tasks.
  arXiv  Detail & Related papers  (2026-04-04T14:51:09Z)
• Paper Summary Attack: Jailbreaking LLMs through LLM Safety Papers [61.57691030102618]
  We propose a novel jailbreaking method, Paper Summary Attack (llmnamePSA)<n>It synthesizes content from either attack-focused or defense-focused LLM safety paper to construct an adversarial prompt template.<n>Experiments show significant vulnerabilities not only in base LLMs, but also in state-of-the-art reasoning model like Deepseek-R1.
  arXiv  Detail & Related papers  (2025-07-17T18:33:50Z)
• AICrypto: A Comprehensive Benchmark for Evaluating Cryptography Capabilities of Large Language Models [31.974963309762913]
  AICrypto is the first comprehensive benchmark designed to evaluate the cryptography capabilities of large language models.<n>The benchmark comprises 135 multiple-choice questions, 150 capture-the-flag challenges, and 18 proof problems.<n>Our evaluation of 17 leading LLMs reveals that state-of-the-art models match or even surpass human experts in memorizing cryptographic concepts.
  arXiv  Detail & Related papers  (2025-07-13T11:11:01Z)
• Improving LLM Agents with Reinforcement Learning on Cryptographic CTF Challenges [0.0]
  'Random-Crypto' is a procedurally generated cryptographic dataset designed to unlock the potential of Reinforcement Learning.<n>We fine-tune a Python tool-augmented Llama-3.1-8B via Group Relative Policy Optimization.<n>The resulting agent achieves a significant improvement in Pass@8 on previously unseen challenges.
  arXiv  Detail & Related papers  (2025-06-01T01:59:52Z)
• CipherBank: Exploring the Boundary of LLM Reasoning Capabilities through Cryptography Challenges [42.16123880046729]
  We introduce CipherBank, a benchmark designed to evaluate the reasoning capabilities of large language models (LLMs) in cryptographic decryption tasks.<n>We evaluate state-of-the-art LLMs on CipherBank, e.g., GPT-4o, DeepSeek-V3, and cutting-edge reasoning-focused models such as o1 and DeepSeek-R1.
  arXiv  Detail & Related papers  (2025-04-27T03:41:17Z)
• Cloning Games, Black Holes and Cryptography [50.022147589030304]
  We introduce a new toolkit for analyzing cloning games.<n>This framework allows us to analyze a new cloning game based on binary phase states.<n>We show that the binary phase variantally optimal bound offers quantitative insights into information scrambling in idealized models of black holes.
  arXiv  Detail & Related papers  (2024-11-07T14:09:32Z)
• Ciphertext Malleability in Lattice-Based KEMs as a Countermeasure to Side Channel Analysis [0.0]
  We discuss a vulnerability due to the ciphertext malleability of some Post-Quantum Cryptography primitives exposed by Ravi et al.<n>We propose a novel countermeasure exploiting the same ciphertext malleability and discuss its practical application to several PQC primitives.
  arXiv  Detail & Related papers  (2024-09-24T14:07:48Z)
• CodeChameleon: Personalized Encryption Framework for Jailbreaking Large Language Models [49.60006012946767]
  We propose CodeChameleon, a novel jailbreak framework based on personalized encryption tactics. We conduct extensive experiments on 7 Large Language Models, achieving state-of-the-art average Attack Success Rate (ASR) Remarkably, our method achieves an 86.6% ASR on GPT-4-1106.
  arXiv  Detail & Related papers  (2024-02-26T16:35:59Z)
• Weak-to-Strong Jailbreaking on Large Language Models [92.52448762164926]
  Large language models (LLMs) are vulnerable to jailbreak attacks.<n>Existing jailbreaking methods are computationally costly.<n>We propose the weak-to-strong jailbreaking attack.
  arXiv  Detail & Related papers  (2024-01-30T18:48:37Z)
• Faster Secure Data Mining via Distributed Homomorphic Encryption [108.77460689459247]
  Homomorphic Encryption (HE) is receiving more and more attention recently for its capability to do computations over the encrypted field. We propose a novel general distributed HE-based data mining framework towards one step of solving the scaling problem. We verify the efficiency and effectiveness of our new framework by testing over various data mining algorithms and benchmark data-sets.
  arXiv  Detail & Related papers  (2020-06-17T18:14:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.