How Do LLMs Read Bug Reports? An Empirical Study of Attention in LLMs for Automated Program Repair

• URL: http://arxiv.org/abs/2607.25873v1
• Date: Tue, 28 Jul 2026 15:38:12 GMT
• Title: How Do LLMs Read Bug Reports? An Empirical Study of Attention in LLMs for Automated Program Repair
• Abstract summary: This paper presents the first empirical study of attention patterns in Large Language Model (LLM)-based program repair.<n>We analyze 319 real-world Python and Java bugs to study how model attention is distributed across bug report sections.<n>We find that successful repairs are characterized by diffused attention across multiple diagnostic components.
• Score: 7.1093539424910155
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Model (LLM)-based Automated Program Repair systems are advancing rapidly, yet their performance remains inconsistent. Even when provided with the same contextual information, an LLM may generate a correct patch for one bug but fail on another closely related bug. Why this happens remains poorly understood, and it is unclear how LLMs prioritize the diverse information in bug reports and whether model attention affects repair success. In this paper, we present the first empirical study of attention patterns in LLM-based program repair, providing interpretable insights into how models process bug reports and where their attention is concentrated during repair. We analyze 319 real-world Python and Java bugs from SWE-bench Verified and Multi-SWE-bench to study (RQ1) how model attention is distributed across bug report sections, (RQ2) how attention patterns within each section differ between successful and unsuccessful repairs, and (RQ3) how these patterns compare to information developers consider important for bug fixing. We find that successful repairs are characterized by diffused attention across multiple diagnostic components such as bug descriptions, stacktraces, and test cases, while failures often exhibit over-localized attention toward metadata such as version information. We further observe that stronger alignment between model attention and developer-identified key sections and phrases is associated with higher repair success. Our results provide the first empirical evidence that attention misallocation is a key factor in LLM-based APR failures, and offer actionable insights for designing more interpretable and reliable future APR systems.

Related papers

• Understanding Agent-Reactive Bugs at the Model-Harness Boundary: An Empirical Study of LLM Agent Issue Reports [9.560342483305371]
  This study is the first empirical study focused on agent-reactive (AR) bugs.<n>Our findings show that many AR bugs manifest as silent errors without well-defined test oracles.<n>These findings point to the need for mechanisms that help users and developers understand the root causes and resolutions of AR bugs.
  arXiv  Detail & Related papers  (2026-07-17T06:53:31Z)
• DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems [48.971606069204825]
  DoVer is an intervention-driven debug framework for large language model (LLM)-based multi-agent systems.<n>It augments hypothesis generation with active verification through targeted interventions.<n>DoVer flips 18-28% of failed trials into successes, achieves up to 16% milestone progress, and validates or refutes 30-60% of failure hypotheses.
  arXiv  Detail & Related papers  (2025-12-07T09:23:48Z)
• InspectCoder: Dynamic Analysis-Enabled Self Repair through interactive LLM-Debugger Collaboration [71.18377595277018]
  Large Language Models (LLMs) frequently generate buggy code with complex logic errors that are challenging to diagnose.<n>We present InspectCoder, the first agentic program repair system that empowers LLMs to actively conduct dynamic analysis via interactive debugger control.
  arXiv  Detail & Related papers  (2025-10-21T06:26:29Z)
• Where LLM Agents Fail and How They can Learn From Failures [62.196870049524364]
  Large Language Model (LLM) agents have shown promise in solving complex, multi-step tasks.<n>They amplify vulnerability to cascading failures, where a single root-cause error propagates through subsequent decisions.<n>Current systems lack a framework that can comprehensively understand agent error in a modular and systemic way.<n>We introduce the AgentErrorTaxonomy, a modular classification of failure modes spanning memory, reflection, planning, action, and system-level operations.
  arXiv  Detail & Related papers  (2025-09-29T18:20:27Z)
• RelRepair: Enhancing Automated Program Repair by Retrieving Relevant Code [11.74568238259256]
  RelRepair retrieves relevant project-specific code to enhance automated program repair.<n>We evaluate RelRepair on two widely studied datasets, Defects4J V1.2 and ManySStuBs4J.
  arXiv  Detail & Related papers  (2025-09-20T14:07:28Z)
• Do AI models help produce verified bug fixes? [62.985237003585674]
  Large Language Models are used to produce corrections to software bugs.<n>This paper investigates how programmers use Large Language Models to complement their own skills.<n>The results are a first step towards a proper role for AI and LLMs in providing guaranteed-correct fixes to program bugs.
  arXiv  Detail & Related papers  (2025-07-21T17:30:16Z)
• Learning Software Bug Reports: A Systematic Literature Review [4.019641745947759]
  Machine learning (ML) aims to automate the understanding, extraction, and correlation of information from bug reports.<n>Despite its growing importance, there has been no comprehensive review in this area.<n>In this paper, we present a systematic literature review covering 1,825 papers, selecting 204 for detailed analysis.
  arXiv  Detail & Related papers  (2025-07-06T15:17:59Z)
• Towards Understanding Bugs in Distributed Training and Inference Frameworks for Large Language Models [7.486731499255164]
  This paper conducts the first large-scale empirical analysis of 308 fixed bugs across three popular distributed training/inference frameworks: DeepSpeed, Megatron-LM, and Colossal-AI.<n>We examine bug symptoms, root causes, bug identification and fixing efforts, and common low-effort fixing strategies.
  arXiv  Detail & Related papers  (2025-06-12T07:24:59Z)
• AutoDetect: Towards a Unified Framework for Automated Weakness Detection in Large Language Models [95.09157454599605]
  Large Language Models (LLMs) are becoming increasingly powerful, but they still exhibit significant but subtle weaknesses.<n>Traditional benchmarking approaches cannot thoroughly pinpoint specific model deficiencies.<n>We introduce a unified framework, AutoDetect, to automatically expose weaknesses in LLMs across various tasks.
  arXiv  Detail & Related papers  (2024-06-24T15:16:45Z)
• A Deep Dive into Large Language Models for Automated Bug Localization and Repair [12.756202755547024]
  Large language models (LLMs) have shown impressive effectiveness in various software engineering tasks, including automated program repair (APR) In this study, we take a deep dive into automated bug fixing utilizing LLMs. This methodological separation of bug localization and fixing using different LLMs enables effective integration of diverse contextual information. Toggle achieves the new state-of-the-art (SOTA) performance on the CodeXGLUE code refinement benchmark.
  arXiv  Detail & Related papers  (2024-04-17T17:48:18Z)
• The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation [93.01964988474755]
  AutoMQM is a prompting technique which asks large language models to identify and categorize errors in translations. We study the impact of labeled data through in-context learning and finetuning. We then evaluate AutoMQM with PaLM-2 models, and we find that it improves performance compared to just prompting for scores.
  arXiv  Detail & Related papers  (2023-08-14T17:17:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.