LLM-Based Detection of Tangled Code Changes for Higher-Quality Method-Level Bug Datasets

• URL: http://arxiv.org/abs/2505.08263v2
• Date: Sat, 19 Jul 2025 17:36:19 GMT
• Title: LLM-Based Detection of Tangled Code Changes for Higher-Quality Method-Level Bug Datasets
• Authors: Md Nahidul Islam Opu, Shaowei Wang, Shaiful Chowdhury,
• Abstract summary: We investigate the utility of Large Language Models for detecting tangled code changes by leveraging both commit messages and method-level code diffs.<n>Our results demonstrate that combining commit messages with code diffs significantly enhances model performance.<n>Applying our approach to 49 open-source projects improves the distributional separability of code metrics between buggy and non-buggy methods.
• Score: 5.191767648600372
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Tangled code changes, commits that conflate unrelated modifications such as bug fixes, refactorings, and enhancements, introduce significant noise into bug datasets and adversely affect the performance of bug prediction models. Addressing this issue at a fine-grained, method-level granularity remains underexplored. This is critical to address, as recent bug prediction models, driven by practitioner demand, are increasingly focusing on finer granularity rather than traditional class- or file-level predictions. This study investigates the utility of Large Language Models (LLMs) for detecting tangled code changes by leveraging both commit messages and method-level code diffs. We formulate the problem as a binary classification task and evaluate multiple prompting strategies, including zero-shot, few-shot, and chain-of-thought prompting, using state-of-the-art proprietary LLMs such as GPT-4o and Gemini-2.0-Flash. Our results demonstrate that combining commit messages with code diffs significantly enhances model performance, with the combined few-shot and chain-of-thought prompting achieving an F1-score of 0.88. Additionally, we explore machine learning models trained on LLM-generated embeddings, where a multi-layer perceptron classifier achieves superior performance (F1-score: 0.906, MCC: 0.807). Applying our approach to 49 open-source projects improves the distributional separability of code metrics between buggy and non-buggy methods, demonstrating the promise of LLMs for method-level commit untangling and potentially contributing to improving the accuracy of future bug prediction models.

Related papers

• DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation [68.19756761027351]
  Diffusion large language models (dLLMs) are compelling alternatives to autoregressive (AR) models.<n>We investigate their denoising processes and reinforcement learning methods.<n>Our work provides deeper insight into the machinery of dLLM generation and offers an effective, diffusion-native RL training framework.
  arXiv  Detail & Related papers  (2025-06-25T17:35:47Z)
• Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free [81.65559031466452]
  We conduct experiments to investigate gating-augmented softmax attention variants.<n>We find that a simple modification-applying a head-specific sigmoid gate after the Scaled Dot-Product Attention (SDPA)-consistently improves performance.
  arXiv  Detail & Related papers  (2025-05-10T17:15:49Z)
• Secret Breach Detection in Source Code with Large Language Models [2.5484785866796833]
  Leaking sensitive information in source code remains a persistent security threat.<n>This work aims to enhance secret detection in source code using large language models (LLMs)<n>We evaluate the feasibility of using fine-tuned, smaller models for local deployment.
  arXiv  Detail & Related papers  (2025-04-26T03:33:14Z)
• iHHO-SMOTe: A Cleansed Approach for Handling Outliers and Reducing Noise to Improve Imbalanced Data Classification [0.0]
  Classifying imbalanced datasets remains a significant challenge in machine learning.<n>Synthetic Minority Over-sampling Technique (SMOTE) generates new instances for the under-represented minority class.<n>A proposed approach, iHHO-SMOTe, addresses the limitations of SMOTE by first cleansing the data from noise points.
  arXiv  Detail & Related papers  (2025-04-17T11:17:53Z)
• IterPref: Focal Preference Learning for Code Generation via Iterative Debugging [28.020886216989872]
  We propose IterPref, a new preference alignment framework for Code LLMs.<n>IterPref explicitly locates error regions and aligns the corresponding tokens via a tailored DPO algorithm.<n>IterPref achieves significant performance gains in code generation and improves on challenging tasks like BigCodeBench.
  arXiv  Detail & Related papers  (2025-03-04T16:56:34Z)
• DECIDER: Leveraging Foundation Model Priors for Improved Model Failure Detection and Explanation [18.77296551727931]
  We propose DECIDER, a novel approach that leverages priors from large language models (LLMs) and vision-language models (VLMs) to detect failures in image models. DECIDER consistently achieves state-of-the-art failure detection performance, significantly outperforming baselines in terms of the overall Matthews correlation coefficient.
  arXiv  Detail & Related papers  (2024-08-01T07:08:11Z)
• What's Wrong with Your Code Generated by Large Language Models? An Extensive Study [80.18342600996601]
  Large language models (LLMs) produce code that is shorter yet more complicated as compared to canonical solutions. We develop a taxonomy of bugs for incorrect codes that includes three categories and 12 sub-categories, and analyze the root cause for common bug types. We propose a novel training-free iterative method that introduces self-critique, enabling LLMs to critique and correct their generated code based on bug types and compiler feedback.
  arXiv  Detail & Related papers  (2024-07-08T17:27:17Z)
• LLMs-as-Instructors: Learning from Errors Toward Automating Model Improvement [93.38736019287224]
  "LLMs-as-Instructors" framework autonomously enhances the training of smaller target models. Inspired by the theory of "Learning from Errors", this framework employs an instructor LLM to meticulously analyze the specific errors within a target model. Within this framework, we implement two strategies: "Learning from Error," which focuses solely on incorrect responses to tailor training data, and "Learning from Error by Contrast", which uses contrastive learning to analyze both correct and incorrect responses for a deeper understanding of errors.
  arXiv  Detail & Related papers  (2024-06-29T17:16:04Z)
• A Deep Dive into Large Language Models for Automated Bug Localization and Repair [12.756202755547024]
  Large language models (LLMs) have shown impressive effectiveness in various software engineering tasks, including automated program repair (APR) In this study, we take a deep dive into automated bug fixing utilizing LLMs. This methodological separation of bug localization and fixing using different LLMs enables effective integration of diverse contextual information. Toggle achieves the new state-of-the-art (SOTA) performance on the CodeXGLUE code refinement benchmark.
  arXiv  Detail & Related papers  (2024-04-17T17:48:18Z)
• Language Models are Better Bug Detector Through Code-Pair Classification [0.26107298043931204]
  In this paper, we propose code-pair classification task in which both the buggy and non-buggy versions are given to the model, and the model identifies the buggy ones. Experiments indicate that an LLM can often pick the buggy from the non-buggy version of the code, and the code-pair classification task is much easier compared to be given a snippet and deciding if and where a bug exists.
  arXiv  Detail & Related papers  (2023-11-14T07:20:57Z)
• Method-Level Bug Severity Prediction using Source Code Metrics and LLMs [0.628122931748758]
  We investigate source code metrics, source code representation using large language models (LLMs), and their combination in predicting bug severity labels. Our results suggest that Decision Tree and Random Forest models outperform other models regarding our several evaluation metrics. CodeBERT finetuning improves the bug severity prediction results significantly in the range of 29%-140% for several evaluation metrics.
  arXiv  Detail & Related papers  (2023-09-06T14:38:07Z)
• LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyond [135.8013388183257]
  We propose a new protocol for inconsistency detection benchmark creation and implement it in a 10-domain benchmark called SummEdits. Most LLMs struggle on SummEdits, with performance close to random chance. The best-performing model, GPT-4, is still 8% below estimated human performance.
  arXiv  Detail & Related papers  (2023-05-23T21:50:06Z)
• To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis [50.31589712761807]
  Large language models (LLMs) are notoriously token-hungry during pre-training, and high-quality text data on the web is approaching its scaling limit for LLMs. We investigate the consequences of repeating pre-training data, revealing that the model is susceptible to overfitting. Second, we examine the key factors contributing to multi-epoch degradation, finding that significant factors include dataset size, model parameters, and training objectives.
  arXiv  Detail & Related papers  (2023-05-22T17:02:15Z)
• Discover, Explanation, Improvement: An Automatic Slice Detection Framework for Natural Language Processing [72.14557106085284]
  slice detection models (SDM) automatically identify underperforming groups of datapoints. This paper proposes a benchmark named "Discover, Explain, improve (DEIM)" for classification NLP tasks. Our evaluation shows that Edisa can accurately select error-prone datapoints with informative semantic features.
  arXiv  Detail & Related papers  (2022-11-08T19:00:00Z)
• Improving Calibration for Long-Tailed Recognition [68.32848696795519]
  We propose two methods to improve calibration and performance in such scenarios. For dataset bias due to different samplers, we propose shifted batch normalization. Our proposed methods set new records on multiple popular long-tailed recognition benchmark datasets.
  arXiv  Detail & Related papers  (2021-04-01T13:55:21Z)
• Meta-Learned Confidence for Few-shot Learning [60.6086305523402]
  A popular transductive inference technique for few-shot metric-based approaches, is to update the prototype of each class with the mean of the most confident query examples. We propose to meta-learn the confidence for each query sample, to assign optimal weights to unlabeled queries. We validate our few-shot learning model with meta-learned confidence on four benchmark datasets.
  arXiv  Detail & Related papers  (2020-02-27T10:22:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.