SCDBench: A Benchmark for LLM-Based Smart Contract Decompilers

• URL: http://arxiv.org/abs/2605.29059v1
• Date: Wed, 27 May 2026 20:08:47 GMT
• Title: SCDBench: A Benchmark for LLM-Based Smart Contract Decompilers
• Abstract summary: We introduce SCDBench, a dataset and benchmark methodology for smart contract decompilation.<n>The dataset contains 600 real-world Solidity contracts with paired bytecode inputs, ground-truth source code, and replayable semantic checkpoints.<n>We evaluate Claude Opus 4.7, GPT-5.3-Codex, and GLM-5 in a zero-shot decompilation setting, including GLM-5 variants with and without extended reasoning.
• Score: 55.39407031861402
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Smart contract decompilation aims to recover high-level source code from bytecode, but evaluating decompilers remains difficult because existing studies use narrow datasets, inconsistent metrics, and limited semantic consistency checks. This gap is increasingly important as large language models (LLMs) begin to generate source-like Solidity that may compile and appear plausible, even when its semantics diverge from the original contract. We introduce SCDBench, a dataset and benchmark methodology for LLM-based smart contract decompilation. The dataset contains 600 real-world Solidity contracts with paired bytecode inputs, ground-truth source code, and replayable semantic checkpoints. SCDBench evaluates decompiler outputs through four cumulative stages: format completeness, compilability, Application Binary Interface (ABI) recovery, and semantic consistency via differential replay. We evaluate Claude Opus 4.7, GPT-5.3-Codex, and GLM-5 in a zero-shot decompilation setting, including GLM-5 variants with and without extended reasoning and a zero-shot compilation-repair setting. The results show that frontier LLMs can often produce structured and compilable Solidity, but achieving semantic consistency remains far from solved: the best-performing frontier model perfectly decompiles only 42/600 contracts. We further show that introducing same-model compilation repair substantially improves performance at modest additional cost. SCDBench establishes a common ground for rigorous, reproducible evaluation and aims to accelerate the development of reliable smart contract decompilers for blockchain security and transparency.

Related papers

• Repository-Level Solidity Code Generation with Large Language Models: From Prompting to Fine-Tuning [10.22195567763592]
  Solidity smart contracts represent a high-stakes domain where generated code must satisfy strict language-level, security, and software-engineering constraints.<n>Existing benchmarks and metrics remain insufficient for repository-level Solidity generation.<n>We introduce SolidityBench, a benchmark of 5,470 repository-level Solidity smart contracts paired with natural language descriptions.
  arXiv  Detail & Related papers  (2026-06-18T09:28:30Z)
• Constraint-Guided Multi-Agent Decompilation for Executable Binary Recovery [47.704311990064554]
  Decompilation is essential for security analysis, malware reverse engineering, and legacy software maintenance.<n>We present a multi-agent framework that transforms decompiled code into re-executable source through Multi-level Constraint-Guided Decompilation (MCGD)<n>Our framework achieves 84-97% re-executability, improving baseline decompiler output by 28-89 percentage points.
  arXiv  Detail & Related papers  (2026-04-27T01:28:11Z)
• RealBench: A Repo-Level Code Generation Benchmark Aligned with Real-World Software Development Practices [54.956760584923295]
  Researchers have made substantial progress using Large Language Models (LLMs) for code generation.<n>However, developers typically write code based on structured designs or specifications rather than raw natural language descriptions.<n>This gap between existing benchmarks and real industry development practices means that current benchmark scores may not accurately reflect how much code generation can help automate development tasks.
  arXiv  Detail & Related papers  (2026-04-24T15:35:54Z)
• ReCUBE: Evaluating Repository-Level Context Utilization in Code Generation [7.907933839674293]
  Large Language Models (LLMs) have emerged as capable coding assistants that operate over larges through either agentic exploration or full-context generation.<n>We introduce ReCUBE, a benchmark in which LLMs reconstruct a masked file within a real-world repository, using all remaining source files, dependency specifications, and documentation as their only source of context.<n>We propose the Caller-Centric Exploration (CCE) toolkit, a set of dependency graph-based tools that can be integrated into agentic frameworks.
  arXiv  Detail & Related papers  (2026-03-26T08:04:15Z)
• RealSec-bench: A Benchmark for Evaluating Secure Code Generation in Real-World Repositories [58.32028251925354]
  Large Language Models (LLMs) have demonstrated remarkable capabilities in code generation, but their proficiency in producing secure code remains a critical, under-explored area.<n>We introduce RealSec-bench, a new benchmark for secure code generation meticulously constructed from real-world, high-risk Java repositories.
  arXiv  Detail & Related papers  (2026-01-30T08:29:01Z)
• Beyond Code Similarity: Benchmarking the Plausibility, Efficiency, and Complexity of LLM-Generated Smart Contracts [3.3672086394822762]
  LLMs produce code with high semantic similarity to real contracts.<n>Only 20% to 26% of zero-shot generations behave identically to ground-truth implementations under testing.<n>Retrieval-Augmented Generation markedly improves performance, boosting functional correctness by up to 45%.
  arXiv  Detail & Related papers  (2025-11-20T10:47:59Z)
• Context-Guided Decompilation: A Step Towards Re-executability [50.71992919223209]
  Binary decompilation plays an important role in software security analysis, reverse engineering and malware understanding.<n>Recent advances in large language models (LLMs) have enabled neural decompilation, but the generated code is typically only semantically plausible.<n>We propose ICL4Decomp, a hybrid decompilation framework that leverages in-context learning (ICL) to guide LLMs toward generating re-executable source code.
  arXiv  Detail & Related papers  (2025-11-03T17:21:39Z)
• D-LiFT: Improving LLM-based Decompiler Backend via Code Quality-driven Fine-tuning [49.16469288280772]
  Decompilers reconstruct human-readable source code from binaries.<n>Despite recent advances, their outputs often suffer from syntactic and semantic errors and remain difficult to read.<n>With the advent of large language models (LLMs), researchers began to explore the potential of LLMs to refine decompiler output.<n>We present D-LIFT, an enhanced decompiler-LLM pipeline with fine-tuned reinforcement learning.
  arXiv  Detail & Related papers  (2025-06-11T19:09:08Z)
• Towards Source Mapping for Zero-Knowledge Smart Contracts: Design and Preliminary Evaluation [9.952399779710044]
  We present a source mapping framework that establishes traceability between Solidity source code, LLVM IR, and zkEVM bytecode within the zkSolc compilation pipeline.<n>We evaluate the framework on a dataset of 50 benchmark contracts and 500 real-world zkSync contracts, observing a mapping accuracy of approximately 97.2% for standard Solidity constructs.
  arXiv  Detail & Related papers  (2025-04-06T01:42:07Z)
• SolBench: A Dataset and Benchmark for Evaluating Functional Correctness in Solidity Code Completion and Repair [51.0686873716938]
  We introduce SolBench, a benchmark for evaluating the functional correctness of Solidity smart contracts generated by code completion models.<n>We propose a Retrieval-Augmented Code Repair framework to verify functional correctness of smart contracts.<n>Results show that code repair and retrieval techniques effectively enhance the correctness of smart contract completion while reducing computational costs.
  arXiv  Detail & Related papers  (2025-03-03T01:55:20Z)
• Augmenting Smart Contract Decompiler Output through Fine-grained Dependency Analysis and LLM-facilitated Semantic Recovery [35.28336403990799]
  SmartHalo is a new framework that enhances decompiler output by combining static analysis (SA) and large language models (LLM)<n>SmartHalo significantly improves the quality of the decompiled code, compared to SOTA decompilers (e.g., Gigahorse)
  arXiv  Detail & Related papers  (2025-01-15T09:04:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.