CyberChainBench: Can AI Agents Secure Smart Contracts Against Real-World On-Chain Vulnerabilities?

• URL: http://arxiv.org/abs/2606.26216v1
• Date: Wed, 24 Jun 2026 17:58:16 GMT
• Title: CyberChainBench: Can AI Agents Secure Smart Contracts Against Real-World On-Chain Vulnerabilities?
• Abstract summary: CyberChainBench is a benchmark for evaluating LLM-based agents on smart contract security.<n>Built from 541 real-world exploit incidents from DeFiHackLabs, the benchmark provides end-to-end on-chain evaluation.<n>Results reveal a clear difficulty gradient: the best configuration scores 37.5% on detection, 43.7% on exploitation, but only 23.4% on patching.
• Score: 28.049089946230513
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present CyberChainBench, a benchmark for evaluating LLM-based agents on smart contract security across three complementary tasks: vulnerability detection, exploit generation, and patch synthesis. Built from 541 real-world exploit incidents from DeFiHackLabs spanning 9 EVM chains, the benchmark provides end-to-end on-chain evaluation where agents interact with historical blockchain state through isolated evaluation environments orchestrated by Harbor, using tools to read code, trace transactions, and validate exploits on mainnet forks. Each case is anchored to a specific block and includes structured ground truth covering vulnerability type, localization, and attacker profit. Exploits are graded by economic impact on historical forks; patches are validated by replaying historical attacks and legitimate transactions as fail-to-pass test oracles on a proxy-upgradeable subset. We define a five-type vulnerability taxonomy and evaluate multiple agent--model configurations. Results reveal a clear difficulty gradient: the best configuration scores 37.5% on detection, 43.7% on exploitation, but only 23.4% on patching, with the top agent (Codex with GPT-5.5) realizing \$57.4M in total exploit profit across the 200-case exploit set at a cost of $2.39 per case.

Related papers

• BenchShield: Formal Model-Backed Instrumentation for Reward Integrity in LLM-Agent Evaluation Infrastructure [47.61632934139226]
  LM-agent benchmarks increasingly function as interactive evaluation infrastructure.<n>This interactivity makes evaluations vulnerable to reward hacking.<n>This paper presents BenchShield, a model-backed instrumentation layer for reward integrity.
  arXiv  Detail & Related papers  (2026-09-10T03:10:58Z)
• Beyond Detection: Agentic Attack Synthesis and Simulation for Smart Contracts [9.850249387268157]
  KASS (Knowledge-Augmented Attack Synthesis and Simulation) is a multi-agent framework for executable smart contract exploit verification.<n>It decomposes automated exploit generation into planning, generation, and testing stages.<n>It produces structured attack plans that document exploitation flows, quantify potential asset losses, and serve as semantic false positive filters for static analysis tools.
  arXiv  Detail & Related papers  (2026-07-17T06:31:49Z)
• ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks? [92.21756459993695]
  Exploitation is a challenging task because it requires low-level program reasoning.<n>Despite its importance and diagnostic value, exploitation remains under-evaluated.<n>We introduce ExploitGym, a large-scale, diverse, realistic benchmark on the exploitation capabilities of AI agents.
  arXiv  Detail & Related papers  (2026-05-11T18:00:14Z)
• GoAT-X: A Graph of Auditing Thoughts for Securing Token Transactions in Cross-Chain Contracts [52.51342355102833]
  Cross-chain bridges, the critical infrastructure of the multi-chain ecosystem, have become a primary target for attackers.<n>Existing defenses, such as bytecode-level static analysis, are ill-equipped to handle the semantic complexity of cross-chain interactions.<n>We propose GoAT-X, a framework that shifts automated cross-chain smart contracts auditing from pattern matching toward systematic first-principles verification.
  arXiv  Detail & Related papers  (2026-04-27T11:34:21Z)
• V2E: Validating Smart Contract Vulnerabilities through Profit-driven Exploit Generation and Execution [52.57923394143007]
  V2E is a new framework for validating whether a reported vulnerability is truly exploitable.<n>V2E automatically generates Proof-of-Concept (PoC) and then assess if the vulnerability could be triggered and incur any financial loss.<n> Evaluation on 264 manually labeled contracts shows that V2E outperforms the baseline approach.
  arXiv  Detail & Related papers  (2026-04-15T08:24:24Z)
• Measuring and Exploiting Confirmation Bias in LLM-Assisted Security Code Review [6.417595678110472]
  We study whether confirmation bias affects LLM-based vulnerability detection, and whether this failure mode can be exploited in software supply-chain attacks.<n>Study 1 quantifies confirmation bias through controlled experiments on 250 CVE vulnerability/patch pairs evaluated across four state-of-the-art models under five framing conditions for the review prompt.<n>Study 2 evaluates exploitability in practice mimicking adversarial pull requests that reintroduce known vulnerabilities while framed as security improvements or urgent functionality fixes via their pull request metadata.
  arXiv  Detail & Related papers  (2026-03-19T10:40:27Z)
• On-Chain Decentralized Learning and Cost-Effective Inference for DeFi Attack Mitigation [4.37137147573556]
  Billions of dollars are lost every year in DeFi platforms by transactions exploiting business logic or accounting vulnerabilities.<n>We present the first decentralized, fully on-chain learning framework that: (i) performs gas-prohibitive computation on Layer-2 to reduce cost, (ii) propagates verified model updates to Layer-1, and (iii) enables gas-bounded low-latency inference inside smart contracts.
  arXiv  Detail & Related papers  (2025-10-15T18:58:34Z)
• BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems [62.17474934536671]
  We introduce the first framework to capture offensive and defensive cyber-capabilities in evolving real-world systems.<n>To capture the vulnerability lifecycle, we define three task types: Detect (detecting a new vulnerability), Exploit (exploiting a specific vulnerability), and Patch (patching a specific vulnerability)<n>We evaluate 8 agents: Claude Code, OpenAI Codex CLI with o3-high and o4-mini, and custom agents with o3-high, GPT-4.1, Gemini 2.5 Pro Preview, Claude 3.7 Sonnet Thinking, and DeepSeek-R1.
  arXiv  Detail & Related papers  (2025-05-21T07:44:52Z)
• Insecurity Through Obscurity: Veiled Vulnerabilities in Closed-Source Contracts [11.609699771118116]
  We present SKANF, a novel bytecode analysis tool tailored for closed-source and obfuscated contracts.<n>SKANF combines control-flow deobfuscation, symbolic execution, and concolic execution based on historical transactions to identify and exploit asset management vulnerabilities.<n>Our evaluation on real-world Maximal Extractable Value (MEV) bots reveals that SKANF detects vulnerabilities in 1,030 contracts and successfully generates exploits for 394 of them, with potential losses of $10.6M.
  arXiv  Detail & Related papers  (2025-04-18T01:22:58Z)
• Smart Contract Fuzzing Towards Profitable Vulnerabilities [10.908512696717724]
  VERITE is a profit-centric smart contract fuzzing framework.<n>It detects profitable vulnerabilities and maximizes the exploited profits.<n>It can extract more than 18 million dollars in total and is significantly better than state-of-the-art fuzzer ITYFUZZ in both detection and exploitation.
  arXiv  Detail & Related papers  (2025-01-15T14:38:18Z)
• ESCORT: Ethereum Smart COntRacTs Vulnerability Detection using Deep Neural Network and Transfer Learning [80.85273827468063]
  Existing machine learning-based vulnerability detection methods are limited and only inspect whether the smart contract is vulnerable. We propose ESCORT, the first Deep Neural Network (DNN)-based vulnerability detection framework for smart contracts. We show that ESCORT achieves an average F1-score of 95% on six vulnerability types and the detection time is 0.02 seconds per contract.
  arXiv  Detail & Related papers  (2021-03-23T15:04:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.