On the Impact of Code Comments for Automated Bug-Fixing: An Empirical Study

• URL: http://arxiv.org/abs/2601.23059v1
• Date: Fri, 30 Jan 2026 15:07:27 GMT
• Title: On the Impact of Code Comments for Automated Bug-Fixing: An Empirical Study
• Abstract summary: Comments may play a critical role in fixing certain types of bugs by providing valuable design and implementation insights.<n>We investigate how the presence or absence of comments, both during training and at inference time, impacts the bug-fixing capabilities of Large Language Models.<n>Our findings show that comments improve ABF accuracy by up to threefold when present in both phases, while training with comments does not degrade performance when instances lack them.
• Score: 3.8571904945643336
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) are increasingly relevant in Software Engineering research and practice, with Automated Bug Fixing (ABF) being one of their key applications. ABF involves transforming a buggy method into its fixed equivalent. A common preprocessing step in ABF involves removing comments from code prior to training. However, we hypothesize that comments may play a critical role in fixing certain types of bugs by providing valuable design and implementation insights. In this study, we investigate how the presence or absence of comments, both during training and at inference time, impacts the bug-fixing capabilities of LLMs. We conduct an empirical evaluation comparing two model families, each evaluated under all combinations of training and inference conditions (with and without comments), and thereby revisiting the common practice of removing comments during training. To address the limited availability of comments in state-of-the-art datasets, we use an LLM to automatically generate comments for methods lacking them. Our findings show that comments improve ABF accuracy by up to threefold when present in both phases, while training with comments does not degrade performance when instances lack them. Additionally, an interpretability analysis identifies that comments detailing method implementation are particularly effective in aiding LLMs to fix bugs accurately.

Related papers

• Beyond the Traceback: Using LLMs for Adaptive Explanations of Programming Errors [52.619793817715326]
  We present a multi-stage crowdsourced study evaluating skill-targeted, LLM-generated Python error messages.<n>We show that while LLM-rewritten messages significantly improved subjective evaluations, these perceived gains did not translate into statistically significant improvements in objective debug performance.<n>This highlights a critical human-AI complementarity gap: explanations that feel better to users do not necessarily make them more effective debuggers.
  arXiv  Detail & Related papers  (2026-08-21T09:14:28Z)
• Rethinking Training Data for Generating Code Review Comments [1.7212147528714599]
  We argue that problematic training instances are not homogeneous and that some limitations stem from deeper issues in the task formulation itself.<n>We derive a taxonomy of misalignment capturing semantic ambiguity, lack of actionability, and context dependence.
  arXiv  Detail & Related papers  (2026-07-28T15:24:30Z)
• Context-Adaptive Requirements Defect Prediction through Human-LLM Collaboration [1.4499356176178066]
  We propose a Human-LLM Collaboration (HLC) approach that treats defect prediction as an adaptive process rather than a static classification task.<n>We evaluate this approach using the weak word smell on the QuRE benchmark of 1,266 annotated Mercedes-Benz requirements.
  arXiv  Detail & Related papers  (2026-01-05T10:00:14Z)
• Exploring the Potential of Large Language Models in Fine-Grained Review Comment Classification [4.61232919707345]
  Large Language Models (LLMs) can classify 17 categories of code review comments.<n>LLMs achieve better accuracy in classifying the five most useful categories.<n>These results suggest that the LLMs could offer a scalable solution for code review analytics.
  arXiv  Detail & Related papers  (2025-08-13T14:07:05Z)
• Do AI models help produce verified bug fixes? [62.985237003585674]
  Large Language Models are used to produce corrections to software bugs.<n>This paper investigates how programmers use Large Language Models to complement their own skills.<n>The results are a first step towards a proper role for AI and LLMs in providing guaranteed-correct fixes to program bugs.
  arXiv  Detail & Related papers  (2025-07-21T17:30:16Z)
• LLM-Safety Evaluations Lack Robustness [58.334290876531036]
  We argue that current safety alignment research efforts for large language models are hindered by many intertwined sources of noise.<n>We propose a set of guidelines for reducing noise and bias in evaluations of future attack and defense papers.
  arXiv  Detail & Related papers  (2025-03-04T12:55:07Z)
• Identifying Inaccurate Descriptions in LLM-generated Code Comments via Test Execution [11.418182511485032]
  We evaluate comments generated by three Large Language Models (LLMs) We propose the concept of document testing, in which a document is verified by using an LLM to generate tests based on the document, running those tests, and observing whether they pass or fail.
  arXiv  Detail & Related papers  (2024-06-21T02:40:34Z)
• Towards Effective Evaluations and Comparisons for LLM Unlearning Methods [97.2995389188179]
  This paper seeks to refine the evaluation of machine unlearning for large language models.<n>It addresses two key challenges -- the robustness of evaluation metrics and the trade-offs between competing goals.
  arXiv  Detail & Related papers  (2024-06-13T14:41:00Z)
• The Right Prompts for the Job: Repair Code-Review Defects with Large Language Model [15.885824575879763]
  Automatic program repair (APR) techniques have the potential to reduce manual efforts in uncovering and repairing program defects during the code review (CR) process. However, the limited accuracy and considerable time costs associated with existing APR approaches hinder their adoption in industrial practice. Recent advancements in Large Language Models (LLMs) have enhanced their ability to comprehend natural and programming languages, enabling them to generate patches based on review comments.
  arXiv  Detail & Related papers  (2023-12-29T06:12:15Z)
• Constructive Large Language Models Alignment with Diverse Feedback [76.9578950893839]
  We introduce Constructive and Diverse Feedback (CDF) as a novel method to enhance large language models alignment. We exploit critique feedback for easy problems, refinement feedback for medium problems, and preference feedback for hard problems. By training our model with this diversified feedback, we achieve enhanced alignment performance while using less training data.
  arXiv  Detail & Related papers  (2023-10-10T09:20:14Z)
• FELM: Benchmarking Factuality Evaluation of Large Language Models [40.78878196872095]
  We introduce a benchmark for Factuality Evaluation of large Language Models, referred to as felm. We collect responses generated from large language models and annotate factuality labels in a fine-grained manner. Our findings reveal that while retrieval aids factuality evaluation, current LLMs are far from satisfactory to faithfully detect factual errors.
  arXiv  Detail & Related papers  (2023-10-01T17:37:31Z)
• CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing [139.77117915309023]
  CRITIC allows large language models to validate and amend their own outputs in a manner similar to human interaction with tools. Comprehensive evaluations involving free-form question answering, mathematical program synthesis, and toxicity reduction demonstrate that CRITIC consistently enhances the performance of LLMs.
  arXiv  Detail & Related papers  (2023-05-19T15:19:44Z)
• Deep Just-In-Time Inconsistency Detection Between Comments and Source Code [51.00904399653609]
  In this paper, we aim to detect whether a comment becomes inconsistent as a result of changes to the corresponding body of code. We develop a deep-learning approach that learns to correlate a comment with code changes. We show the usefulness of our approach by combining it with a comment update model to build a more comprehensive automatic comment maintenance system.
  arXiv  Detail & Related papers  (2020-10-04T16:49:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.