Crash Report Enhancement with Large Language Models: An Empirical Study

• URL: http://arxiv.org/abs/2509.13535v1
• Date: Tue, 16 Sep 2025 21:02:57 GMT
• Title: Crash Report Enhancement with Large Language Models: An Empirical Study
• Abstract summary: We examine whether large language models can enhance crash reports by adding fault locations, root-cause explanations, and repair suggestions.<n>On a dataset of 492 real-world crash reports, LLM-enhanced reports improve Top-1 problem-localization accuracy from 10.6% to 40.2-43.1%.<n>Both our manual evaluations and LLM-as-a-judge assessment show that Agentic-LLM delivers stronger root-cause explanations and more actionable repair guidance.
• Score: 21.842377922082104
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Crash reports are central to software maintenance, yet many lack the diagnostic detail developers need to debug efficiently. We examine whether large language models can enhance crash reports by adding fault locations, root-cause explanations, and repair suggestions. We study two enhancement strategies: Direct-LLM, a single-shot approach that uses stack-trace context, and Agentic-LLM, an iterative approach that explores the repository for additional evidence. On a dataset of 492 real-world crash reports, LLM-enhanced reports improve Top-1 problem-localization accuracy from 10.6% (original reports) to 40.2-43.1%, and produce suggested fixes that closely resemble developer patches (CodeBLEU around 56-57%). Both our manual evaluations and LLM-as-a-judge assessment show that Agentic-LLM delivers stronger root-cause explanations and more actionable repair guidance. A user study with 16 participants further confirms that enhanced reports make crashes easier to understand and resolve, with the largest improvement in repair guidance. These results indicate that supplying LLMs with stack traces and repository code yields enhanced crash reports that are substantially more useful for debugging.

Related papers

• Integrating Crash Report Mining and LLMs for Bug Localization and Repair: An Industrial Report [7.575395371401797]
  Analyzing crash-report bugs in large-scale industrial software systems requires substantial maintenance effort.<n>We show that the best-performing configuration localizes up to 71% and correctly repairs 52% of crash bugs on the full dataset.
  arXiv  Detail & Related papers  (2026-09-03T21:04:40Z)
• How Do LLMs Read Bug Reports? An Empirical Study of Attention in LLMs for Automated Program Repair [7.1093539424910155]
  This paper presents the first empirical study of attention patterns in Large Language Model (LLM)-based program repair.<n>We analyze 319 real-world Python and Java bugs to study how model attention is distributed across bug report sections.<n>We find that successful repairs are characterized by diffused attention across multiple diagnostic components.
  arXiv  Detail & Related papers  (2026-07-28T15:38:12Z)
• Writing Bug Reports for Software Repair Agents: What Information Matters Most? [45.10027008889631]
  We study what makes a bug report agent-ready.<n>We focus on the 441 issues representing bug reports.<n>Our results suggest that agentic-first reports benefit most from information that narrows the agent's search and repair space.
  arXiv  Detail & Related papers  (2026-07-10T16:00:57Z)
• Outrunning LLM Cutoffs: A Live Kernel Crash Resolution Benchmark for All [57.23434868678603]
  Live-kBench is an evaluation framework for self-evolving benchmarks that scrapes and evaluates agents on freshly discovered kernel bugs.<n> kEnv is an agent-agnostic crash-resolution environment for kernel compilation, execution, and feedback.<n>Using kEnv, we benchmark three state-of-the-art agents, showing that they resolve 74% of crashes on the first attempt.
  arXiv  Detail & Related papers  (2026-02-02T19:06:15Z)
• DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems [48.971606069204825]
  DoVer is an intervention-driven debug framework for large language model (LLM)-based multi-agent systems.<n>It augments hypothesis generation with active verification through targeted interventions.<n>DoVer flips 18-28% of failed trials into successes, achieves up to 16% milestone progress, and validates or refutes 30-60% of failure hypotheses.
  arXiv  Detail & Related papers  (2025-12-07T09:23:48Z)
• InspectCoder: Dynamic Analysis-Enabled Self Repair through interactive LLM-Debugger Collaboration [71.18377595277018]
  Large Language Models (LLMs) frequently generate buggy code with complex logic errors that are challenging to diagnose.<n>We present InspectCoder, the first agentic program repair system that empowers LLMs to actively conduct dynamic analysis via interactive debugger control.
  arXiv  Detail & Related papers  (2025-10-21T06:26:29Z)
• Where LLM Agents Fail and How They can Learn From Failures [62.196870049524364]
  Large Language Model (LLM) agents have shown promise in solving complex, multi-step tasks.<n>They amplify vulnerability to cascading failures, where a single root-cause error propagates through subsequent decisions.<n>Current systems lack a framework that can comprehensively understand agent error in a modular and systemic way.<n>We introduce the AgentErrorTaxonomy, a modular classification of failure modes spanning memory, reflection, planning, action, and system-level operations.
  arXiv  Detail & Related papers  (2025-09-29T18:20:27Z)
• Eigen-1: Adaptive Multi-Agent Refinement with Monitor-Based RAG for Scientific Reasoning [53.45095336430027]
  We develop a unified framework that combines implicit retrieval and structured collaboration.<n>On Humanity's Last Exam (HLE) Bio/Chem Gold, our framework achieves 48.3% accuracy.<n>Results on SuperGPQA and TRQA confirm robustness across domains.
  arXiv  Detail & Related papers  (2025-09-25T14:05:55Z)
• RelRepair: Enhancing Automated Program Repair by Retrieving Relevant Code [11.74568238259256]
  RelRepair retrieves relevant project-specific code to enhance automated program repair.<n>We evaluate RelRepair on two widely studied datasets, Defects4J V1.2 and ManySStuBs4J.
  arXiv  Detail & Related papers  (2025-09-20T14:07:28Z)
• RepoDebug: Repository-Level Multi-Task and Multi-Language Debugging Evaluation of Large Language Models [49.83481415540291]
  Large Language Models (LLMs) have exhibited significant proficiency in code debug.<n>This paper introduces Repo Debug, a multi-task and multi-language repository-level code debug dataset.<n>We conduct evaluation experiments on 10 LLMs, where Claude 3.5 Sonnect, the best-performing model, still cannot perform well in repository-level debug.
  arXiv  Detail & Related papers  (2025-09-04T10:13:21Z)
• Verifying the Verifiers: Unveiling Pitfalls and Potentials in Fact Verifiers [59.168391398830515]
  We evaluate 12 pre-trained LLMs and one specialized fact-verifier, using a collection of examples from 14 fact-checking benchmarks.<n>We highlight the importance of addressing annotation errors and ambiguity in datasets.<n> frontier LLMs with few-shot in-context examples, often overlooked in previous works, achieve top-tier performance.
  arXiv  Detail & Related papers  (2025-06-16T10:32:10Z)
• An Empirical Study on the Capability of LLMs in Decomposing Bug Reports [9.544728752295269]
  This study investigates whether large language models (LLMs) can assist developers in automatically decomposing complex bug reports into smaller, self-contained units.<n>We conducted an empirical study on 127 resolved privacy-related bug reports collected from Apache Jira.
  arXiv  Detail & Related papers  (2025-04-29T16:29:12Z)
• Code Change Intention, Development Artifact and History Vulnerability: Putting Them Together for Vulnerability Fix Detection by LLM [13.278153690972243]
  VulFixMiner and CoLeFunDa focus solely on code changes, neglecting essential context from development artifacts.<n>We propose LLM4VFD, a novel framework that leverages Large Language Models (LLMs) enhanced with Chain-of-Thought reasoning and In-Context Learning.
  arXiv  Detail & Related papers  (2025-01-24T23:40:03Z)
• Enhancing IR-based Fault Localization using Large Language Models [5.032687557488094]
  This paper enhances Fault Localization (IRFL) by categorizing bug reports based on programming entities, stack traces, and natural language text.<n>To address inaccuracies in queries, we introduce a user and conversational-based query reformulation approach, termed LLmiRQ+.<n> Evaluation on 46 projects with 6,340 bug reports yields an MRR of 0.6770 and MAP of 0.5118, surpassing seven state-of-the-art IRFL techniques.
  arXiv  Detail & Related papers  (2024-12-04T22:47:51Z)
• The Impact Of Bug Localization Based on Crash Report Mining: A Developers' Perspective [7.952391285456257]
  We report our experience of using an approach for grouping crash reports and finding buggy code on a weekly basis for 18 months. The approach investigated in this study correctly suggested the buggy file most of the time -- the approach's precision was around 80%.
  arXiv  Detail & Related papers  (2024-03-16T01:23:01Z)
• DebugBench: Evaluating Debugging Capability of Large Language Models [80.73121177868357]
  DebugBench is a benchmark for Large Language Models (LLMs) It covers four major bug categories and 18 minor types in C++, Java, and Python. We evaluate two commercial and four open-source models in a zero-shot scenario.
  arXiv  Detail & Related papers  (2024-01-09T15:46:38Z)
• Exploring Large Language Models in Resolving Environment-Related Crash Bugs: Localizing and Repairing [36.4673637256627]
  Large language models (LLMs) have shown promise in software engineering tasks.<n>We conduct the first comprehensive study to assess the capability of LLMs in resolving real-world environment crash bugs.<n>Our findings reveal that localization is the primary challenge for resolving code crashes, while repair poses a greater challenge for environment crashes.
  arXiv  Detail & Related papers  (2023-12-16T13:41:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.