Decompile-Bench: Million-Scale Binary-Source Function Pairs for Real-World Binary Decompilation

• URL: http://arxiv.org/abs/2505.12668v1
• Date: Mon, 19 May 2025 03:34:33 GMT
• Title: Decompile-Bench: Million-Scale Binary-Source Function Pairs for Real-World Binary Decompilation
• Authors: Hanzhuo Tan, Xiaolong Tian, Hanrui Qi, Jiaming Liu, Zuchen Gao, Siyi Wang, Qi Luo, Jing Li, Yuqun Zhang,
• Abstract summary: Decompile-Bench is the first open-source dataset comprising two million binary-source function pairs condensed from 100 million collected function pairs.<n>For the evaluation purposes, we developed a benchmark Decompile-Bench-Eval including manually crafted binaries from the well-established HumanEval and MBPP.<n>We find that fine-tuning with Decompile-Bench causes a 20% improvement over previous benchmarks in terms of the re-executability rate.
• Score: 12.983487033256448
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in LLM-based decompilers have been shown effective to convert low-level binaries into human-readable source code. However, there still lacks a comprehensive benchmark that provides large-scale binary-source function pairs, which is critical for advancing the LLM decompilation technology. Creating accurate binary-source mappings incurs severe issues caused by complex compilation settings and widespread function inlining that obscure the correspondence between binaries and their original source code. Previous efforts have either relied on used contest-style benchmarks, synthetic binary-source mappings that diverge significantly from the mappings in real world, or partially matched binaries with only code lines or variable names, compromising the effectiveness of analyzing the binary functionality. To alleviate these issues, we introduce Decompile-Bench, the first open-source dataset comprising two million binary-source function pairs condensed from 100 million collected function pairs, i.e., 450GB of binaries compiled from permissively licensed GitHub projects. For the evaluation purposes, we also developed a benchmark Decompile-Bench-Eval including manually crafted binaries from the well-established HumanEval and MBPP, alongside the compiled GitHub repositories released after 2025 to mitigate data leakage issues. We further explore commonly-used evaluation metrics to provide a thorough assessment of the studied LLM decompilers and find that fine-tuning with Decompile-Bench causes a 20% improvement over previous benchmarks in terms of the re-executability rate. Our code and data has been released in HuggingFace and Github. https://github.com/albertan017/LLM4Decompile

Related papers

• D-LiFT: Improving LLM-based Decompiler Backend via Code Quality-driven Fine-tuning [49.16469288280772]
  We present D-LiFT, an automated decompiler backend that harnesses and trains LLMs to improve the quality of decompiled code via reinforcement learning (RL)<n>D-LiFT adheres to a key principle for enhancing the quality of decompiled code: textitpreserving accuracy while improving readability.<n>Central to D-LiFT, we propose D-SCORE, an integrated quality assessment system to score the decompiled code from multiple aspects.
  arXiv  Detail & Related papers  (2025-06-11T19:09:08Z)
• An Empirical Study on the Effectiveness of Large Language Models for Binary Code Understanding [50.17907898478795]
  This work proposes a benchmark to evaluate the effectiveness of Large Language Models (LLMs) in real-world reverse engineering scenarios.<n>Our evaluations reveal that existing LLMs can understand binary code to a certain extent, thereby improving the efficiency of binary code analysis.
  arXiv  Detail & Related papers  (2025-04-30T17:02:06Z)
• ReF Decompile: Relabeling and Function Call Enhanced Decompile [50.86228893636785]
  The goal of decompilation is to convert compiled low-level code (e.g., assembly code) back into high-level programming languages.<n>This task supports various reverse engineering applications, such as vulnerability identification, malware analysis, and legacy software migration.
  arXiv  Detail & Related papers  (2025-02-17T12:38:57Z)
• Assemblage: Automatic Binary Dataset Construction for Machine Learning [35.674339346299654]
  Assemblage is a cloud-based distributed system that crawls, configures, and builds Windows PE binaries. We have run Assemblage on AWS over the past year, producing 890k Windows PE and 428k Linux ELF binaries across 29 configurations.
  arXiv  Detail & Related papers  (2024-05-07T04:10:01Z)
• How Far Have We Gone in Binary Code Understanding Using Large Language Models [51.527805834378974]
  We propose a benchmark to evaluate the effectiveness of Large Language Models (LLMs) in binary code understanding. Our evaluations reveal that existing LLMs can understand binary code to a certain extent, thereby improving the efficiency of binary code analysis.
  arXiv  Detail & Related papers  (2024-04-15T14:44:08Z)
• LLM4Decompile: Decompiling Binary Code with Large Language Models [10.346311290153398]
  Decompilation aims to convert binary code to high-level source code, but traditional tools like Ghidra often produce results difficult to read and execute. We propose LLM4Decompile, the first and largest open-source LLM series (1.3B to 33B) trained to decompile binary code. The resulting models significantly outperform GPT-4o and Ghidra on the HumanEval and ExeBench benchmarks by over 100% in terms of re-executability rate.
  arXiv  Detail & Related papers  (2024-03-08T13:10:59Z)
• BinaryAI: Binary Software Composition Analysis via Intelligent Binary Source Code Matching [8.655595404611821]
  We introduce BinaryAI, a novel binary-to-source SCA technique with two-phase binary source code matching to capture both syntactic and semantic code features. Our experimental results demonstrate the superior performance of BinaryAI in terms of binary source code matching and the downstream SCA task.
  arXiv  Detail & Related papers  (2024-01-20T07:57:57Z)
• RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation [96.75695811963242]
  RepoCoder is a framework to streamline the repository-level code completion process. It incorporates a similarity-based retriever and a pre-trained code language model. It consistently outperforms the vanilla retrieval-augmented code completion approach.
  arXiv  Detail & Related papers  (2023-03-22T13:54:46Z)
• Extending Source Code Pre-Trained Language Models to Summarise Decompiled Binaries [4.0484792045035505]
  We extend large pre-trained language models of source code to summarise decompiled binary functions. We investigate the impact of input and data properties on the performance of such models. BinT5 achieves the state-of-the-art BLEU-4 score of 60.83, 58.82, and 44.21 for summarising source, decompiled, and synthetically stripped decompiled code.
  arXiv  Detail & Related papers  (2023-01-04T16:56:33Z)
• Boosting Neural Networks to Decompile Optimized Binaries [13.255618541522436]
  Decompilation aims to transform a low-level program language (LPL) into its functionally-equivalent high-level program language (HPL) We propose a novel learning-based approach named NeurDP, that targets compiler-optimized binaries.
  arXiv  Detail & Related papers  (2023-01-03T06:45:54Z)
• Towards Accurate Binary Neural Networks via Modeling Contextual Dependencies [52.691032025163175]
  Existing Binary Neural Networks (BNNs) operate mainly on local convolutions with binarization function. We present new designs of binary neural modules, which enables leading binary neural modules by a large margin.
  arXiv  Detail & Related papers  (2022-09-03T11:51:04Z)
• Improving type information inferred by decompilers with supervised machine learning [0.0]
  In software reverse engineering, decompilation is the process of recovering source code from binary files. We build different classification models capable of inferring the high-level type returned by functions. Our system is able to predict function return types with a 79.1% F1-measure, whereas the best decompiler obtains a 30% F1-measure.
  arXiv  Detail & Related papers  (2021-01-19T11:45:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.