CODEFUSE-DEBENCH: An Empirical Study on Readability, Recompilability, and Functionality

• URL: http://arxiv.org/abs/2605.29490v1
• Date: Thu, 28 May 2026 07:17:53 GMT
• Title: CODEFUSE-DEBENCH: An Empirical Study on Readability, Recompilability, and Functionality
• Abstract summary: We propose a reusability-driven evaluation paradigm that measures decompiler quality along three dimensions: readability, recompilability, and functionality.<n>We present DEBENCH, the first automated framework for multidimensional decompilation evaluation.
• Score: 14.354022325641088
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Binary decompilation aims to recover binaries into high-level source code, but existing evaluations mainly rely on syntactic similarity or single-axis readability metrics, which fail to capture practical reusability. We propose a reusability-driven evaluation paradigm that measures decompiler quality along three orthogonal dimensions: readability, recompilability, and functionality. We present DEBENCH, the first automated framework for multidimensional decompilation evaluation. DEBENCH contains 240 atomic test functions, organized into 8 source files and compiled into 640 binaries. It combines LLM-as-judge readability scoring with URAF (18 sub-dimensions), iterative compile-and-repair under a fixed 50-iteration budget, and Frida-based differential dynamic tracing at the program, function, and instruction levels. We evaluate five mainstream decompilers and three repair LLMs. Our study reveals four findings. First, the reusability cliff is steep: the best decompiler-LLM pair reaches 22.3% Exact+Partial program-level behavioral overlap but only 1.2% exact stdout match, nearly 50 points below recompilability. Second, settings that maximize readability do not maximize functionality: -O3 yields the lowest readability but the highest functionality, and Clang gives lower readability than GCC but 2.6x higher functionality. Third, cross-decompiler variation at the functional level is 20x, far larger than the 1.6x cross-LLM variation, showing that progress depends more on decompiler engines than larger repair models. Fourth, failures fall into three categories: syntactic noise, type-system collapse (about 19% of repair errors), and irreversible upstream losses such as ARM64 relocation idioms and C++ ABI features.

Related papers

• CHISEL-ing Back Source Code with AI-enabled Iterative Recovery [7.946261291068861]
  Decompilation aims to recover high-level, compilable, and semantically equivalent code from binaries.<n>Traditional decompilers produce pseudo-C that is difficult to read and does not compile, while the recent LLM-assisted approaches generate readable, but semantically incorrect code.<n>We present CHISEL, a test suite-free framework to iteratively recover source code from Ghidra-derived pseudo-C.
  arXiv  Detail & Related papers  (2026-08-28T06:49:21Z)
• REFORGE: A Method for Benchmarking LLMs' Reverse Engineering Capabilities in Decompiled Binary Function Naming [0.0]
  Large language models (LLMs) are increasingly applied to reverse-engineering tasks.<n>Claims about their capability, however, outpace our ability to measure it.<n>This paper presents Reforge, a provenance-tracked pipeline that constructs function-level ground truth from C source.
  arXiv  Detail & Related papers  (2026-07-07T23:17:03Z)
• Constraint-Guided Multi-Agent Decompilation for Executable Binary Recovery [47.704311990064554]
  Decompilation is essential for security analysis, malware reverse engineering, and legacy software maintenance.<n>We present a multi-agent framework that transforms decompiled code into re-executable source through Multi-level Constraint-Guided Decompilation (MCGD)<n>Our framework achieves 84-97% re-executability, improving baseline decompiler output by 28-89 percentage points.
  arXiv  Detail & Related papers  (2026-04-27T01:28:11Z)
• SPARC: Scenario Planning and Reasoning for Automated C Unit Test Generation [1.0010193170880752]
  We introduce a neuro-symbolic, scenario-based framework that bridges the gap between high-level program intent and the rigid syntactic constraints of pointer arithmetic and manual memory management.<n>We evaluate it on 59 real-world and algorithmic subjects, where it outperforms the vanilla prompt generation baseline by 31.36% in line coverage, 26.01% in branch coverage, and 20.78% in mutation score, matching or exceeding the symbolic execution tool KLEE.
  arXiv  Detail & Related papers  (2026-02-18T18:09:03Z)
• HELIOS: Hierarchical Graph Abstraction for Structure-Aware LLM Decompilation [11.110675371854988]
  textscHELIOS is a framework that reframes binary decompilation as a structured reasoning task.<n>textscHELIOS is a practical building block for reverse engineering in security settings.
  arXiv  Detail & Related papers  (2026-01-21T02:37:33Z)
• QiMeng-NeuComBack: Self-Evolving Translation from IR to Assembly Code [52.66657751895655]
  Large Language Models (LLMs) offer a compelling new paradigm: Neural Compilation.<n>This paper introduces NeuComBack, a novel benchmark dataset specifically designed for IR-to-assembly compilation.<n>We propose a self-evolving prompt optimization method that enables LLMs to evolve their internal prompt strategies.
  arXiv  Detail & Related papers  (2025-11-03T03:20:26Z)
• Improving Compiler Bug Isolation by Leveraging Large Language Models [14.679589768900621]
  We propose an innovative compiler bug isolation approach named AutoCBI.<n>We evaluate AutoCBI against state-of-the-art approaches (DiWi, RecBi and FuseFL) on 120 real-world bugs from the widely-used GCC and LLVM compilers.<n>Specifically, AutoCBI isolates 66.67%/69.23%, 300%/340%, and 100%/57.14% more bugs than RecBi, DiWi, and FuseFL, respectively, in the Top-1 ranked results for GCC/LLVM.
  arXiv  Detail & Related papers  (2025-06-21T09:09:30Z)
• D-LiFT: Improving LLM-based Decompiler Backend via Code Quality-driven Fine-tuning [49.16469288280772]
  Decompilers reconstruct human-readable source code from binaries.<n>Despite recent advances, their outputs often suffer from syntactic and semantic errors and remain difficult to read.<n>With the advent of large language models (LLMs), researchers began to explore the potential of LLMs to refine decompiler output.<n>We present D-LIFT, an enhanced decompiler-LLM pipeline with fine-tuned reinforcement learning.
  arXiv  Detail & Related papers  (2025-06-11T19:09:08Z)
• EquiBench: Benchmarking Large Language Models' Reasoning about Program Semantics via Equivalence Checking [58.15568681219339]
  We introduce EquiBench, a new benchmark for evaluating large language models (LLMs)<n>This task directly tests a model's ability to reason about program semantics.<n>We evaluate 19 state-of-the-art LLMs and find that in the most challenging categories, the best accuracies are 63.8% and 76.2%, only modestly above the 50% random baseline.
  arXiv  Detail & Related papers  (2025-02-18T02:54:25Z)
• ReF Decompile: Relabeling and Function Call Enhanced Decompile [50.86228893636785]
  The goal of decompilation is to convert compiled low-level code (e.g., assembly code) back into high-level programming languages.<n>This task supports various reverse engineering applications, such as vulnerability identification, malware analysis, and legacy software migration.
  arXiv  Detail & Related papers  (2025-02-17T12:38:57Z)
• ReGAL: Refactoring Programs to Discover Generalizable Abstractions [59.05769810380928]
  Generalizable Abstraction Learning (ReGAL) is a method for learning a library of reusable functions via codeization. We find that the shared function libraries discovered by ReGAL make programs easier to predict across diverse domains. For CodeLlama-13B, ReGAL results in absolute accuracy increases of 11.5% on LOGO, 26.1% on date understanding, and 8.1% on TextCraft, outperforming GPT-3.5 in two of three domains.
  arXiv  Detail & Related papers  (2024-01-29T18:45:30Z)
• HDCC: A Hyperdimensional Computing compiler for classification on embedded systems and high-performance computing [58.720142291102135]
  This work introduces the name compiler, the first open-source compiler that translates high-level descriptions of HDC classification methods into optimized C code. name is designed like a modern compiler, featuring an intuitive and descriptive input language, an intermediate representation (IR), and a retargetable backend. To substantiate these claims, we conducted experiments with HDCC on several of the most popular datasets in the HDC literature.
  arXiv  Detail & Related papers  (2023-04-24T19:16:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.