CryptanalysisBench: Can LLMs do Cryptanalysis?
Abstract Overview
This paper introduces CryptanalysisBench, a benchmark comprising 191 cryptanalysis tasks across six families of cryptographic primitives, primarily drawn from NIST standardization competitions. The benchmark is structured into three tiers: Tier 1 for schemes with known practical breaks, Tier 2 for schemes lacking practical attacks (evaluated at full and scaled-down strengths), and a challenge set of frontier production primitives. Tasks are rigorously evaluated using automated security games where models must submit complete attack scripts. Assessing five frontier large language models, the study details both aggregate success rates and qualitative insights into whether successes stem from memorized bugs or genuine cryptanalytic derivation.
Novelty
The novelty of this work resides in its large-scale, automated verification of LLM capabilities on authentic cryptographic schemes rather than simplified toy ciphers or CTF problems. Furthermore, it surfaces genuinely novel discoveries generated autonomously by the models, including an undocumented full key-recovery attack on the SpoC AEAD and a flaw in the published CCA-security proof of KINDI.
Results
Evaluations show that frontier models solve between 65.3% and 85.7% of Tier 1 schemes, with Claude Mythos 5 demonstrating the highest performance. While models struggle significantly more on full-strength Tier 2 algorithms, they still achieve notable wins by exploiting implementation defects, underspecified designs, and scaling artifacts in weakened variants. Expanding the available time budget from 2 hours to 16 hours allows models to synthesize more complex attacks, significantly boosting solver resolution despite persistent limitations in autonomous execution capabilities.
Key Points
- CryptanalysisBench challenges language models with 191 real-world cryptographic tasks verified entirely via automated security games.
- Models successfully exploit 65-86% of Tier 1 algorithms but experience sharp performance drop-offs on Tier 2 full-strength configurations.
- The analysis reveals independent, previously unknown cryptanalytic breaks on SpoC and KINDI, alongside many algorithm implementation flaws.