Language Models are Better Bug Detector Through Code-Pair Classification

• URL: http://arxiv.org/abs/2311.07957v2
• Date: Sun, 28 Jan 2024 02:43:40 GMT
• Title: Language Models are Better Bug Detector Through Code-Pair Classification
• Authors: Kamel Alrashedy, Ahmed Binjahlan
• Abstract summary: In this paper, we propose code-pair classification task in which both the buggy and non-buggy versions are given to the model, and the model identifies the buggy ones. Experiments indicate that an LLM can often pick the buggy from the non-buggy version of the code, and the code-pair classification task is much easier compared to be given a snippet and deciding if and where a bug exists.
• Score: 0.26107298043931204
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) such as GPT-3.5 and CodeLlama are powerful models for code generation and understanding. Fine-tuning these models comes with a high computational cost and requires a large labeled dataset. Alternatively, in-context learning techniques allow models to learn downstream tasks with only a few examples. Recently, researchers have shown how in-context learning performs well in bug detection and repair. In this paper, we propose code-pair classification task in which both the buggy and non-buggy versions are given to the model, and the model identifies the buggy ones. We evaluate our task in real-world dataset of bug detection and two most powerful LLMs. Our experiments indicate that an LLM can often pick the buggy from the non-buggy version of the code, and the code-pair classification task is much easier compared to be given a snippet and deciding if and where a bug exists.

Related papers

• Does Your Neural Code Completion Model Use My Code? A Membership Inference Approach [69.38352966504401]
  We investigate the legal and ethical issues of current neural code completion models. We tailor a membership inference approach (termed CodeMI) that was originally crafted for classification tasks. We evaluate the effectiveness of this adapted approach across a diverse array of neural code completion models.
  arXiv  Detail & Related papers  (2024-04-22T15:54:53Z)
• Language Models for Text Classification: Is In-Context Learning Enough? [54.869097980761595]
  Recent foundational language models have shown state-of-the-art performance in many NLP tasks in zero- and few-shot settings. An advantage of these models over more standard approaches is the ability to understand instructions written in natural language (prompts) This makes them suitable for addressing text classification problems for domains with limited amounts of annotated instances.
  arXiv  Detail & Related papers  (2024-03-26T12:47:39Z)
• DebugBench: Evaluating Debugging Capability of Large Language Models [80.73121177868357]
  DebugBench is a benchmark for Large Language Models (LLMs) It covers four major bug categories and 18 minor types in C++, Java, and Python. We evaluate two commercial and four open-source models in a zero-shot scenario.
  arXiv  Detail & Related papers  (2024-01-09T15:46:38Z)
• Large Language Models of Code Fail at Completing Code with Potential Bugs [30.80172644795715]
  We study the buggy-code completion problem inspired by real-time code suggestion. We find that the presence of potential bugs significantly degrades the generation performance of the high-performing Code-LLMs.
  arXiv  Detail & Related papers  (2023-06-06T06:35:27Z)
• WELL: Applying Bug Detectors to Bug Localization via Weakly Supervised Learning [37.09621161662761]
  This paper proposes a WEakly supervised bug LocaLization (WELL) method to train a bug localization model. With CodeBERT finetuned on the buggy-or-not binary labeled data, WELL can address bug localization in a weakly supervised manner.
  arXiv  Detail & Related papers  (2023-05-27T06:34:26Z)
• Enriching Source Code with Contextual Data for Code Completion Models: An Empirical Study [4.438873396405334]
  We aim to answer whether making code easier to understand through using contextual data improves the performance of pre-trained code language models for the task of code completion. For comments, we find that the models perform better in the presence of multi-line comments.
  arXiv  Detail & Related papers  (2023-04-24T17:09:14Z)
• CodeExp: Explanatory Code Document Generation [94.43677536210465]
  Existing code-to-text generation models produce only high-level summaries of code. We conduct a human study to identify the criteria for high-quality explanatory docstring for code. We present a multi-stage fine-tuning strategy and baseline models for the task.
  arXiv  Detail & Related papers  (2022-11-25T18:05:44Z)
• Evaluating few shot and Contrastive learning Methods for Code Clone Detection [5.1623866691702744]
  Code Clone Detection is a software engineering task that is used for plagiarism detection, code search, and code comprehension. Deep learning-based models have achieved an F1 score (a metric used to assess classifiers) of $sim$95% on the CodeXGLUE benchmark. No previous study evaluates the generalizability of these models where a limited amount of annotated data is available.
  arXiv  Detail & Related papers  (2022-04-15T15:01:55Z)
• DapStep: Deep Assignee Prediction for Stack Trace Error rePresentation [61.99379022383108]
  We propose new deep learning models to solve the bug triage problem. The models are based on a bidirectional recurrent neural network with attention and on a convolutional neural network. To improve the quality of ranking, we propose using additional information from version control system annotations.
  arXiv  Detail & Related papers  (2022-01-14T00:16:57Z)
• Can Machines Read Coding Manuals Yet? -- A Benchmark for Building Better Language Models for Code Understanding [3.98345038769576]
  We derive a set of benchmarks that assess code understanding based on tasks such as predicting the best answer to a question in a forum post. We evaluate the performance of current state-of-the-art language models on these tasks and show that there is a significant improvement on each task from fine tuning.
  arXiv  Detail & Related papers  (2021-09-15T17:42:44Z)
• Overcoming Classifier Imbalance for Long-tail Object Detection with Balanced Group Softmax [88.11979569564427]
  We provide the first systematic analysis on the underperformance of state-of-the-art models in front of long-tail distribution. We propose a novel balanced group softmax (BAGS) module for balancing the classifiers within the detection frameworks through group-wise training. Extensive experiments on the very recent long-tail large vocabulary object recognition benchmark LVIS show that our proposed BAGS significantly improves the performance of detectors.
  arXiv  Detail & Related papers  (2020-06-18T10:24:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.