Understanding Agent-Reactive Bugs at the Model-Harness Boundary: An Empirical Study of LLM Agent Issue Reports

• URL: http://arxiv.org/abs/2607.15684v1
• Date: Fri, 17 Jul 2026 06:53:31 GMT
• Title: Understanding Agent-Reactive Bugs at the Model-Harness Boundary: An Empirical Study of LLM Agent Issue Reports
• Abstract summary: This study is the first empirical study focused on agent-reactive (AR) bugs.<n>Our findings show that many AR bugs manifest as silent errors without well-defined test oracles.<n>These findings point to the need for mechanisms that help users and developers understand the root causes and resolutions of AR bugs.
• Score: 9.560342483305371
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: LLM agents span command-line interfaces (e.g., Codex) and agent frameworks (e.g., LangChain), integrating backend LLMs with harness code that parses model outputs, controls agent loops, and manages context. Both the harness and LLM-generated responses jointly shape an agent's execution. This architecture gives rise to bugs that cannot be readily understood by inspecting either component alone, because some bugs occur only when a particular LLM response elicits an abnormal reaction from the agent. Prior empirical studies of agent bugs have largely attributed failures either to limited model capabilities or to harness-side defects, such as outdated APIs and configuration misalignment, without characterizing these AR bugs. We conduct the first empirical study focused on agent-reactive (AR) bugs. Through manual analysis of 255 bug reports from Codex, Gemini-CLI, LangChain, and CrewAI, we construct a two-axis taxonomy covering observable symptoms and the LLM behaviors that trigger them. Our findings show that many AR bugs manifest as silent errors without well-defined test oracles, which makes detection difficult. The stochasticity of LLM responses further complicates bug reproduction. We additionally examine fixes proposed by users and implemented by developers. This analysis exposes a mismatch: users frequently advocate harness-side guardrails, whereas developers may attribute the issue to the LLM or respond slowly to user-proposed fixes. These findings point to the need for mechanisms that help users and developers understand the root causes and resolutions of AR bugs. Overall, the study highlights challenges specific to LLM agents and motivates the design of test oracles, reproduction support, and fault-localization techniques for AR bugs.

Related papers

• How Do LLMs Read Bug Reports? An Empirical Study of Attention in LLMs for Automated Program Repair [7.1093539424910155]
  This paper presents the first empirical study of attention patterns in Large Language Model (LLM)-based program repair.<n>We analyze 319 real-world Python and Java bugs to study how model attention is distributed across bug report sections.<n>We find that successful repairs are characterized by diffused attention across multiple diagnostic components.
  arXiv  Detail & Related papers  (2026-07-28T15:38:12Z)
• FAMA: Failure-Aware Meta-Agentic Framework for Open-Source LLMs in Interactive Tool Use Environments [60.3427704389541]
  Large Language Models are being increasingly deployed as the decision-making core of autonomous agents.<n>Yet, in conversational benchmarks, these agents frequently fail due to the cascading effects of incorrect decision-making.<n>We present the Failure-Aware Meta-Agentic framework to tackle these challenges.
  arXiv  Detail & Related papers  (2026-04-28T02:21:53Z)
• SelfHeal: Empirical Fix Pattern Analysis and Bug Repair in LLM Agents [3.8743350688734988]
  We study buggy posts and code snippets from Stack Overflow, GitHub, and HuggingFace forums.<n>We introduce AgentDefect, the first benchmark dataset for bugs in LLM agents.<n>We present SelfHeal, a multi-agent system designed to fix bugs in LLM agents.
  arXiv  Detail & Related papers  (2026-04-20T01:28:15Z)
• When Agents Fail: A Comprehensive Study of Bugs in LLM Agents with Automated Labeling [3.0882445204950013]
  We present the first comprehensive study of bug types, root causes, and effects in LLM agent-based software.<n>We collected and analyzed 1,187 bug-related posts and code snippets from Stack Overflow, GitHub, and Hugging Face forums.<n>We built a ReAct agent named BugReAct, equipped with adequate external tools to determine whether it can detect and annotate the bugs in our dataset.
  arXiv  Detail & Related papers  (2026-01-21T18:13:10Z)
• InspectCoder: Dynamic Analysis-Enabled Self Repair through interactive LLM-Debugger Collaboration [71.18377595277018]
  Large Language Models (LLMs) frequently generate buggy code with complex logic errors that are challenging to diagnose.<n>We present InspectCoder, the first agentic program repair system that empowers LLMs to actively conduct dynamic analysis via interactive debugger control.
  arXiv  Detail & Related papers  (2025-10-21T06:26:29Z)
• Where LLM Agents Fail and How They can Learn From Failures [62.196870049524364]
  Large Language Model (LLM) agents have shown promise in solving complex, multi-step tasks.<n>They amplify vulnerability to cascading failures, where a single root-cause error propagates through subsequent decisions.<n>Current systems lack a framework that can comprehensively understand agent error in a modular and systemic way.<n>We introduce the AgentErrorTaxonomy, a modular classification of failure modes spanning memory, reflection, planning, action, and system-level operations.
  arXiv  Detail & Related papers  (2025-09-29T18:20:27Z)
• Towards Automated Error Discovery: A Study in Conversational AI [48.735443116662026]
  We introduce Automated Error Discovery, a framework for detecting and defining errors in conversational AI.<n>We also propose SEEED (Soft Clustering Extended-Based Error Detection), as an encoder-based approach to its implementation.
  arXiv  Detail & Related papers  (2025-09-13T14:53:22Z)
• Do AI models help produce verified bug fixes? [62.985237003585674]
  Large Language Models are used to produce corrections to software bugs.<n>This paper investigates how programmers use Large Language Models to complement their own skills.<n>The results are a first step towards a proper role for AI and LLMs in providing guaranteed-correct fixes to program bugs.
  arXiv  Detail & Related papers  (2025-07-21T17:30:16Z)
• An Empirical Study on the Capability of LLMs in Decomposing Bug Reports [9.544728752295269]
  This study investigates whether large language models (LLMs) can assist developers in automatically decomposing complex bug reports into smaller, self-contained units.<n>We conducted an empirical study on 127 resolved privacy-related bug reports collected from Apache Jira.
  arXiv  Detail & Related papers  (2025-04-29T16:29:12Z)
• PATCH: Empowering Large Language Model with Programmer-Intent Guidance and Collaborative-Behavior Simulation for Automatic Bug Fixing [34.768989900184636]
  Bug fixing holds significant importance in software development and maintenance.<n>Recent research has made substantial strides in exploring the potential of large language models (LLMs) for automatically resolving software bugs.
  arXiv  Detail & Related papers  (2025-01-27T15:43:04Z)
• ExploraCoder: Advancing code generation for multiple unseen APIs via planning and chained exploration [70.26807758443675]
  ExploraCoder is a training-free framework that empowers large language models to invoke unseen APIs in code solution.<n> Experimental results demonstrate that ExploraCoder significantly improves performance for models lacking prior API knowledge.
  arXiv  Detail & Related papers  (2024-12-06T19:00:15Z)
• ThinkNote: Enhancing Knowledge Integration and Utilization of Large Language Models via Constructivist Cognition Modeling [55.21641515545307]
  Large Language Models (LLMs) have demonstrated strong performance across a wide range of NLP tasks.<n>They often exhibit suboptimal behaviors and inconsistencies when exposed to unfamiliar external information.<n>We propose ThinkNote, a novel framework that enhances the external knowledge utilization of LLMs.
  arXiv  Detail & Related papers  (2024-02-21T06:04:53Z)
• ReEval: Automatic Hallucination Evaluation for Retrieval-Augmented Large Language Models via Transferable Adversarial Attacks [91.55895047448249]
  This paper presents ReEval, an LLM-based framework using prompt chaining to perturb the original evidence for generating new test cases. We implement ReEval using ChatGPT and evaluate the resulting variants of two popular open-domain QA datasets. Our generated data is human-readable and useful to trigger hallucination in large language models.
  arXiv  Detail & Related papers  (2023-10-19T06:37:32Z)
• Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback [127.75419038610455]
  Large language models (LLMs) are able to generate human-like, fluent responses for many downstream tasks. This paper proposes a LLM-Augmenter system, which augments a black-box LLM with a set of plug-and-play modules.
  arXiv  Detail & Related papers  (2023-02-24T18:48:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.