Can We Enhance Bug Report Quality Using LLMs?: An Empirical Study of LLM-Based Bug Report Generation

• URL: http://arxiv.org/abs/2504.18804v1
• Date: Sat, 26 Apr 2025 05:15:53 GMT
• Title: Can We Enhance Bug Report Quality Using LLMs?: An Empirical Study of LLM-Based Bug Report Generation
• Authors: Jagrit Acharya, Gouri Ginde,
• Abstract summary: In this paper, we explore whether instruction fine-tuned Large Language Models (LLMs) can automatically transform casual, unstructured bug reports into high-quality, structured bug reports adhering to a standard template.<n>We evaluate three open-source instruction-tuned LLMs (emphQwen 2.5, Mistral, and Llama 3.2) against ChatGPT-4o, measuring performance on established metrics such as CTQRS, ROUGE, METEOR, and SBERT.<n>Our experiments show that fine-tuned Qwen 2.5 achieves a CTQRS score of textbf77%,
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Bug reports contain the information developers need to triage and fix software bugs. However, unclear, incomplete, or ambiguous information may lead to delays and excessive manual effort spent on bug triage and resolution. In this paper, we explore whether Instruction fine-tuned Large Language Models (LLMs) can automatically transform casual, unstructured bug reports into high-quality, structured bug reports adhering to a standard template. We evaluate three open-source instruction-tuned LLMs (\emph{Qwen 2.5, Mistral, and Llama 3.2}) against ChatGPT-4o, measuring performance on established metrics such as CTQRS, ROUGE, METEOR, and SBERT. Our experiments show that fine-tuned Qwen 2.5 achieves a CTQRS score of \textbf{77%}, outperforming both fine-tuned Mistral (\textbf{71%}), Llama 3.2 (\textbf{63%}) and ChatGPT in 3-shot learning (\textbf{75%}). Further analysis reveals that Llama 3.2 shows higher accuracy of detecting missing fields particularly Expected Behavior and Actual Behavior, while Qwen 2.5 demonstrates superior performance in capturing Steps-to-Reproduce, with an F1 score of 76%. Additional testing of the models on other popular projects (e.g., Eclipse, GCC) demonstrates that our approach generalizes well, achieving up to \textbf{70%} CTQRS in unseen projects' bug reports. These findings highlight the potential of instruction fine-tuning in automating structured bug report generation, reducing manual effort for developers and streamlining the software maintenance process.

Related papers

• Automated Bug Report Prioritization in Large Open-Source Projects [3.9134031118910264]
  We propose a novel approach to automated bug prioritization based on the natural language text of the bug reports.<n>We conduct topic modeling using a variant of LDA called TopicMiner-MTM and text classification with the BERT large language model.<n> Experimental results using an existing reference dataset containing 85,156 bug reports of the Eclipse Platform project indicate that we outperform existing approaches in terms of Accuracy, Precision, Recall, and F1-measure of the bug report priority prediction.
  arXiv  Detail & Related papers  (2025-04-22T13:57:48Z)
• LLPut: Investigating Large Language Models for Bug Report-Based Input Generation [0.0]
  Failure-inducing inputs play a crucial role in diagnosing and analyzing software bugs.<n>Prior research has leveraged various Natural Language Processing (NLP) techniques for automated input extraction.<n>With the advent of Large Language Models (LLMs), an important research question arises: how effectively can generative LLMs extract failure-inducing inputs from bug reports?
  arXiv  Detail & Related papers  (2025-03-26T14:25:01Z)
• Improved IR-based Bug Localization with Intelligent Relevance Feedback [2.9312156642007294]
  Software bugs pose a significant challenge during development and maintenance, and practitioners spend nearly 50% of their time dealing with bugs. Many existing techniques adopt Information Retrieval (IR) to localize a reported bug using textual and semantic relevance between bug reports and source code. We present a novel technique for bug localization - BRaIn - that addresses the contextual gaps by assessing the relevance between bug reports and code.
  arXiv  Detail & Related papers  (2025-01-17T20:29:38Z)
• MQM-APE: Toward High-Quality Error Annotation Predictors with Automatic Post-Editing in LLM Translation Evaluators [53.91199933655421]
  Large Language Models (LLMs) have shown significant potential as judges for Machine Translation (MT) quality assessment. We introduce a universal and training-free framework, $textbfMQM-APE, based on the idea of filtering out non-impactful errors. Experiments show that our approach consistently improves both the reliability and quality of error spans against GEMBA-MQM.
  arXiv  Detail & Related papers  (2024-09-22T06:43:40Z)
• DebugBench: Evaluating Debugging Capability of Large Language Models [80.73121177868357]
  DebugBench is a benchmark for Large Language Models (LLMs) It covers four major bug categories and 18 minor types in C++, Java, and Python. We evaluate two commercial and four open-source models in a zero-shot scenario.
  arXiv  Detail & Related papers  (2024-01-09T15:46:38Z)
• The Earth is Flat? Unveiling Factual Errors in Large Language Models [89.94270049334479]
  Large Language Models (LLMs) like ChatGPT are in various applications due to their extensive knowledge from pre-training and fine-tuning. Despite this, they are prone to generating factual and commonsense errors, raising concerns in critical areas like healthcare, journalism, and education. We introduce a novel, automatic testing framework, FactChecker, aimed at uncovering factual inaccuracies in LLMs.
  arXiv  Detail & Related papers  (2024-01-01T14:02:27Z)
• Using Developer Discussions to Guide Fixing Bugs in Software [51.00904399653609]
  We propose using bug report discussions, which are available before the task is performed and are also naturally occurring, avoiding the need for additional information from developers. We demonstrate that various forms of natural language context derived from such discussions can aid bug-fixing, even leading to improved performance over using commit messages corresponding to the oracle bug-fixing commits.
  arXiv  Detail & Related papers  (2022-11-11T16:37:33Z)
• Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproduction [14.444294152595429]
  The number of tests added in open source repositories due to issues was about 28% of the corresponding project test suite size. We propose LIBRO, a framework that uses Large Language Models (LLMs), which have been shown to be capable of performing code-related tasks. Our evaluation of LIBRO shows that, on the widely studied Defects4J benchmark, LIBRO can generate failure reproducing test cases for 33% of all studied cases.
  arXiv  Detail & Related papers  (2022-09-23T10:50:47Z)
• Automatic Classification of Bug Reports Based on Multiple Text Information and Reports' Intention [37.67372105858311]
  This paper proposes a new automatic classification method for bug reports. The innovation is that when categorizing bug reports, in addition to using the text information of the report, the intention of the report is also considered. Our proposed method achieves better performance and its F-Measure achieves from 87.3% to 95.5%.
  arXiv  Detail & Related papers  (2022-08-02T06:44:51Z)
• S3M: Siamese Stack (Trace) Similarity Measure [55.58269472099399]
  We present S3M -- the first approach to computing stack trace similarity based on deep learning. It is based on a biLSTM encoder and a fully-connected classifier to compute similarity. Our experiments demonstrate the superiority of our approach over the state-of-the-art on both open-sourced data and a private JetBrains dataset.
  arXiv  Detail & Related papers  (2021-03-18T21:10:41Z)
• TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task [80.38130122127882]
  TACRED is one of the largest, most widely used crowdsourced datasets in Relation Extraction (RE) In this paper, we investigate the questions: Have we reached a performance ceiling or is there still room for improvement? We find that label errors account for 8% absolute F1 test error, and that more than 50% of the examples need to be relabeled.
  arXiv  Detail & Related papers  (2020-04-30T15:07:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.