A Large Scale Study of AI-based Binary Function Similarity Detection Techniques for Security Researchers and Practitioners

• URL: http://arxiv.org/abs/2511.01180v1
• Date: Mon, 03 Nov 2025 03:10:25 GMT
• Title: A Large Scale Study of AI-based Binary Function Similarity Detection Techniques for Security Researchers and Practitioners
• Abstract summary: Binary Function Similarity Detection (BFSD) is a foundational technique in software security.<n>Recent advances in AI-based BFSD tools have led to significant performance improvements.<n>We present the first large-scale empirical study of AI-based BFSD tools to address these gaps.
• Score: 20.470574733047442
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Binary Function Similarity Detection (BFSD) is a foundational technique in software security, underpinning a wide range of applications including vulnerability detection, malware analysis. Recent advances in AI-based BFSD tools have led to significant performance improvements. However, existing evaluations of these tools suffer from three key limitations: a lack of in-depth analysis of performance-influencing factors, an absence of realistic application analysis, and reliance on small-scale or low-quality datasets. In this paper, we present the first large-scale empirical study of AI-based BFSD tools to address these gaps. We construct two high-quality and diverse datasets: BinAtlas, comprising 12,453 binaries and over 7 million functions for capability evaluation; and BinAres, containing 12,291 binaries and 54 real-world 1-day vulnerabilities for evaluating vulnerability detection performance in practical IoT firmware settings. Using these datasets, we evaluate nine representative BFSD tools, analyze the challenges and limitations of existing BFSD tools, and investigate the consistency among BFSD tools. We also propose an actionable strategy for combining BFSD tools to enhance overall performance (an improvement of 13.4%). Our study not only advances the practical adoption of BFSD tools but also provides valuable resources and insights to guide future research in scalable and automated binary similarity detection.

Related papers

• Understanding Binary Code Similarity for Real-World Vulnerability Detection: A Large-Scale Empirical Study [15.569141472526502]
  We conduct a large-scale study of vulnerability detection across 60,000 firmware images from 200 vendors using BCSD.<n>We examine the influence of four key factors -- vulnerable function versions, vulnerability search space, function sizes, and compilation toolchains on BCSD performance.<n>A TPL-aware, two-stage search process significantly enhances accuracy, improving MRR by 18.5% by limiting the search space.
  arXiv  Detail & Related papers  (2026-06-27T11:26:28Z)
• From Insight to Action: A Novel Framework for Interpretability-Guided Data Selection in Large Language Models [73.72877445629383]
  Interpretability-Guided Data Selection (IGDS) is a framework that first identifies causal task features through frequency recall and interventional filtering.<n>We validate IGDS on mathematical reasoning, summarization, and translation tasks within Gemma-2, LLaMA-3.1, and Qwen3 models.
  arXiv  Detail & Related papers  (2026-04-28T03:16:24Z)
• AIDABench: AI Data Analytics Benchmark [62.45908988324612]
  AIDABench is a benchmark for evaluating AI systems on complex data analytics tasks in an end-to-end manner.<n>AIDABench encompasses 600+ diverse document analysis tasks across three core capability dimensions: question answering, data visualization, and file generation.<n>We evaluate 11 state-of-the-art models on AIDABench, spanning both proprietary (e.g., Claude Sonnet 4.5, Gemini 3 Pro Preview) and open-source (e.g., Qwen3-Max-2026-01-23-Thinking) families.
  arXiv  Detail & Related papers  (2026-02-27T08:58:05Z)
• A Large Scale Empirical Analysis on the Adherence Gap between Standards and Tools in SBOM [54.38424417079265]
  A Software Bill of Materials (SBOM) is a machine-readable artifact that organizes software information.<n>Following standards, organizations have developed tools for generating and utilizing SBOMs.<n>This paper presents the first large-scale, two-stage empirical analysis of the adherence gap, using our automated evaluation framework, SAP.
  arXiv  Detail & Related papers  (2026-01-09T08:26:05Z)
• Ensembling Large Language Models for Code Vulnerability Detection: An Empirical Evaluation [69.8237598448941]
  This study investigates the potential of ensemble learning to enhance the performance of Large Language Models (LLMs) in source code vulnerability detection.<n>We propose Dynamic Gated Stacking (DGS), a Stacking variant tailored for vulnerability detection.
  arXiv  Detail & Related papers  (2025-09-16T03:48:22Z)
• Revisiting Third-Party Library Detection: A Ground Truth Dataset and Its Implications Across Security Tasks [12.328841506102963]
  We present the first large-scale empirical study of ten state-of-the-art TPL detection techniques across over 6,000 apps.<n>Our evaluation exposes tool fragility to R8-era transformations, weak version discrimination, inaccurate correspondence of candidate libraries, difficulty in generalizing similarity thresholds, and prohibitive runtime/memory overheads at scale.
  arXiv  Detail & Related papers  (2025-09-04T10:48:02Z)
• RoHOI: Robustness Benchmark for Human-Object Interaction Detection [84.78366452133514]
  Human-Object Interaction (HOI) detection is crucial for robot-human assistance, enabling context-aware support.<n>We introduce the first benchmark for HOI detection, evaluating model resilience under diverse challenges.<n>Our benchmark, RoHOI, includes 20 corruption types based on the HICO-DET and V-COCO datasets and a new robustness-focused metric.
  arXiv  Detail & Related papers  (2025-07-12T01:58:04Z)
• Advancing Tool-Augmented Large Language Models via Meta-Verification and Reflection Learning [63.2198957755528]
  We propose Tool-MVR, a novel Tool-Augmented LLM that achieves comprehensive System 2 reasoning through two key innovations.<n>Specifically, we first introduce Multi-Agent Meta-Verification (MAMV), a systematic pipeline that rigorously validates APIs, queries, and reasoning trajectories.<n>Second, we propose Exploration-based Reflection Learning (EXPLORE), which enhances tool reflection capabilities by leveraging tool feedback.
  arXiv  Detail & Related papers  (2025-06-05T04:35:49Z)
• An Empirical Analysis of Vulnerability Detection Tools for Solidity Smart Contracts Using Line Level Manually Annotated Vulnerabilities [5.357551358237259]
  This paper provides an empirical evaluation of automated vulnerability analysis tools specifically designed for Solidity smart contracts.<n>We conducted an assessment using an annotated dataset of 2,182 instances we manually annotated with line-level vulnerability labels.<n>We identified a set of 3 tools that, combined, achieve up to 76.78% found vulnerabilities taking less than one minute to run.
  arXiv  Detail & Related papers  (2025-05-21T17:01:18Z)
• Understanding the AI-powered Binary Code Similarity Detection [41.39226562321616]
  AI-powered binary code similarity detection (BinSD) has been widely applied to program analysis. It is difficult to quantitatively understand to what extent the BinSD problem has been solved, especially in realworld applications. We present a systematic evaluation of state-of-the-art AI-powered BinSD approaches.
  arXiv  Detail & Related papers  (2024-10-10T02:13:01Z)
• Underwater Object Detection in the Era of Artificial Intelligence: Current, Challenge, and Future [119.88454942558485]
  Underwater object detection (UOD) aims to identify and localise objects in underwater images or videos. In recent years, artificial intelligence (AI) based methods, especially deep learning methods, have shown promising performance in UOD.
  arXiv  Detail & Related papers  (2024-10-08T00:25:33Z)
• SAFE: Advancing Large Language Models in Leveraging Semantic and Syntactic Relationships for Software Vulnerability Detection [23.7268575752712]
  Software vulnerabilities (SVs) have emerged as a prevalent and critical concern for safety-critical security systems. We propose a novel framework that enhances the capability of large language models to learn and utilize semantic and syntactic relationships from source code data for SVD.
  arXiv  Detail & Related papers  (2024-09-02T00:49:02Z)
• Vulnerability Detection with Code Language Models: How Far Are We? [40.455600722638906]
  PrimeVul is a new dataset for training and evaluating code LMs for vulnerability detection. It incorporates a novel set of data labeling techniques that achieve comparable label accuracy to human-verified benchmarks. It also implements a rigorous data de-duplication and chronological data splitting strategy to mitigate data leakage issues.
  arXiv  Detail & Related papers  (2024-03-27T14:34:29Z)
• Distributed intelligence on the Edge-to-Cloud Continuum: A systematic literature review [62.997667081978825]
  This review aims at providing a comprehensive vision of the main state-of-the-art libraries and frameworks for machine learning and data analytics available today. The main simulation, emulation, deployment systems, and testbeds for experimental research on the Edge-to-Cloud Continuum available today are also surveyed.
  arXiv  Detail & Related papers  (2022-04-29T08:06:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.