CITADEL: Context Similarity Based Deep Learning Framework Bug Finding

• URL: http://arxiv.org/abs/2406.12196v3
• Date: Fri, 13 Sep 2024 07:51:57 GMT
• Title: CITADEL: Context Similarity Based Deep Learning Framework Bug Finding
• Authors: Xiaoyu Zhang, Juan Zhai, Shiqing Ma, Shiwei Wang, Chao Shen,
• Abstract summary: Existing deep learning (DL) framework testing tools have limited coverage on bug types. We propose Citadel, a method that accelerates the finding of bugs in terms of efficiency and effectiveness.
• Score: 36.34154201748415
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: With deep learning (DL) technology becoming an integral part of the new intelligent software, tools of DL framework testing and bug-finding are in high demand. Existing DL framework testing tools have limited coverage on bug types. For example, they lack the capability of finding performance bugs, which are critical for DL model training and inference regarding performance, economics, and the environment. This problem is challenging due to the difficulty of getting test oracles of performance bugs. Moreover, existing tools are inefficient, generating hundreds of test cases with few trigger bugs. In this paper, we propose Citadel, a method that accelerates the finding of bugs in terms of efficiency and effectiveness. We observe that many DL framework bugs are similar due to the similarity of operators and algorithms belonging to the same family (e.g., Conv2D and Conv3D). Orthogonal to existing bug-finding tools, Citadel aims to find new bugs that are similar to reported ones that have known test oracles. It works by first collecting existing bug reports and identifying problematic APIs. Citadel defines context similarity to measure the similarity of DL framework API pairs and automatically generates test cases with oracles for APIs that are similar to the problematic APIs in existing bug reports. Citadel respectively covers 1,436 PyTorch and 5,380 TensorFlow APIs and effectively detects 77 and 74 API bugs, many of which, e.g., 11 performance bugs, cannot be detected by existing tools. Moreover, a remarkable 35.40% of the test cases generated by Citadel can trigger bugs, which significantly transcends the state-of-the-art method (3.90%).

Related papers

• Reinforcement Learning-Based REST API Testing with Multi-Coverage [4.127886193201882]
  MUCOREST is a novel Reinforcement Learning (RL)-based API testing approach that leverages Q-learning to maximize code coverage and output coverage. MUCOREST significantly outperforms state-of-the-art API testing approaches by 11.6-261.1% in the number of discovered API bugs.
  arXiv  Detail & Related papers  (2024-10-20T14:20:23Z)
• KGym: A Platform and Dataset to Benchmark Large Language Models on Linux Kernel Crash Resolution [59.20933707301566]
  Large Language Models (LLMs) are consistently improving at increasingly realistic software engineering (SE) tasks. In real-world software stacks, significant SE effort is spent developing foundational system software like the Linux kernel. To evaluate if ML models are useful while developing such large-scale systems-level software, we introduce kGym and kBench.
  arXiv  Detail & Related papers  (2024-07-02T21:44:22Z)
• DebugBench: Evaluating Debugging Capability of Large Language Models [80.73121177868357]
  DebugBench is a benchmark for Large Language Models (LLMs) It covers four major bug categories and 18 minor types in C++, Java, and Python. We evaluate two commercial and four open-source models in a zero-shot scenario.
  arXiv  Detail & Related papers  (2024-01-09T15:46:38Z)
• Automated Bug Generation in the era of Large Language Models [6.0770779409377775]
  BugFarm transforms arbitrary code into multiple complex bugs. A comprehensive evaluation of 435k+ bugs from over 1.9M mutants generated by BUGFARM.
  arXiv  Detail & Related papers  (2023-10-03T20:01:51Z)
• PreciseBugCollector: Extensible, Executable and Precise Bug-fix Collection [8.79879909193717]
  We introduce PreciseBugCollector, a precise, multi-language bug collection approach. It is based on two novel components: a bug tracker to map the repositories with external bug repositories to trace bug type information, and a bug injector to generate project-specific bugs. To date, PreciseBugCollector comprises 1057818 bugs extracted from 2968 open-source projects.
  arXiv  Detail & Related papers  (2023-09-12T13:47:44Z)
• An Analysis of Bugs In Persistent Memory Application [0.0]
  We evaluate an open-sourced automatic bug detector tool (i.e. AGAMOTTO) to test NVM level hashing PM application. Our faithful validation tool able to discovered 65 new NVM level hashing bugs on PMDK library. We will propose a Deep-Q Learning search algorithm over the PM-Aware search algorithm to improve the searching strategy efficiently.
  arXiv  Detail & Related papers  (2023-07-19T23:12:01Z)
• Prompting Is All You Need: Automated Android Bug Replay with Large Language Models [28.69675481931385]
  We propose AdbGPT, a new lightweight approach to automatically reproduce the bugs from bug reports through prompt engineering. AdbGPT leverages few-shot learning and chain-of-thought reasoning to elicit human knowledge and logical reasoning from LLMs. Our evaluations demonstrate the effectiveness and efficiency of our AdbGPT to reproduce 81.3% of bug reports in 253.6 seconds.
  arXiv  Detail & Related papers  (2023-06-03T03:03:52Z)
• Using Developer Discussions to Guide Fixing Bugs in Software [51.00904399653609]
  We propose using bug report discussions, which are available before the task is performed and are also naturally occurring, avoiding the need for additional information from developers. We demonstrate that various forms of natural language context derived from such discussions can aid bug-fixing, even leading to improved performance over using commit messages corresponding to the oracle bug-fixing commits.
  arXiv  Detail & Related papers  (2022-11-11T16:37:33Z)
• BigIssue: A Realistic Bug Localization Benchmark [89.8240118116093]
  BigIssue is a benchmark for realistic bug localization. We provide a general benchmark with a diversity of real and synthetic Java bugs. We hope to advance the state of the art in bug localization, in turn improving APR performance and increasing its applicability to the modern development cycle.
  arXiv  Detail & Related papers  (2022-07-21T20:17:53Z)
• D2A: A Dataset Built for AI-Based Vulnerability Detection Methods Using Differential Analysis [55.15995704119158]
  We propose D2A, a differential analysis based approach to label issues reported by static analysis tools. We use D2A to generate a large labeled dataset to train models for vulnerability identification.
  arXiv  Detail & Related papers  (2021-02-16T07:46:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.