TrajAudit: Automated Failure Diagnosis for Agentic Coding Systems

• URL: http://arxiv.org/abs/2605.26563v1
• Date: Tue, 26 May 2026 05:24:37 GMT
• Title: TrajAudit: Automated Failure Diagnosis for Agentic Coding Systems
• Abstract summary: Agentic systems have been widely studied to automate software engineering jobs such as bug fixing.<n>Existing automated failure diagnosis approaches leverage task execution trajectories, yet their effectiveness degrades as trajectory length and complexity increase.<n>For repository-level coding tasks specifically, trajectories are laden with noise, such as redundant program structure and verbose code context.<n>We propose TrajAudit, the first failure diagnosis framework for repository-level coding trajectories.
• Score: 18.833806809109536
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Agentic systems have been widely studied to automate software engineering jobs such as bug fixing. As these systems increasingly tackle complex tasks, understanding where and why they fail becomes essential for iterative refinement and operational reliability. Existing automated failure diagnosis approaches leverage task execution trajectories, yet their effectiveness degrades substantially as trajectory length and complexity increase. For repository-level coding tasks specifically, trajectories are laden with noise, such as redundant program structure and verbose code context. Moreover, these trajectories are very long, while long-context reasoning remains a known weakness of LLMs. To address these two challenges, we propose TrajAudit, the first failure diagnosis framework for repository-level coding trajectories. TrajAudit employs an investigator agent supported by two modules: one filters failure-irrelevant information through pattern matching and keyword detection, and the other generates a preliminary diagnosis from test failure reports as prior knowledge, helping the agent handle noisy long contexts. The investigator agent can further invoke tools to retrieve filtered content on demand, ensuring that critical information is preserved while noise is minimized. We also introduce RootSE, a benchmark of 93 real-world agentic failure instances sourced from software maintenance tasks, representing the most complex trajectory diagnosis benchmark to date. Experiments on RootSE show that TrajAudit outperforms all existing baselines by over 24.4 percentage points in localization accuracy, while reducing token consumption by at least 18%, demonstrating its practical effectiveness. We hope this work draws community attention to failure management in agentic software engineering and provides a foundational resource for future research.

Related papers

• Meta-Agent: From Task Descriptions to Verified Multi-Agent Systems [36.485119937780524]
  We present Meta-Agent, a two-phase framework that automatically constructs and executes specialized multi-agent systems.<n>We introduce a three-level error attribution mechanism that distinguishes local, upstream, and structural failures.<n>Experiments against strong multi-agent baselines and ablation studies demonstrate consistent improvements in task success rate, error recovery, and workflow stability.
  arXiv  Detail & Related papers  (2026-05-24T19:38:54Z)
• Trace2Skill: Verifier-Guided Skill Evolution for Long-Context EDA Agents [0.3733676450456031]
  We present Trace2Skill, a test-time scaling framework that improves a hardware agent without RTL-specialized model fine-tuning.<n>Rather than training a new model or only sampling more candidate solutions, Trace2Skill treats the agent's natural-language skill as an evolvable policy.<n>It mines repeated rollout traces for success and failure modes, converts them into dense diagnostics and oracle lessons, and uses an oracle, mutator, and selector loop to produce task-specific skills.
  arXiv  Detail & Related papers  (2026-05-20T23:10:49Z)
• Dynamic analysis enhances issue resolution [53.50448142467294]
  DAIRA (Dynamic Analysis-enhanced Issue Resolution Agent) is an automated repair framework that embeds dynamic analysis into the agent's reasoning cycle.<n> Driven by a Test Tracing-Driven methodology, DAIRA utilizes lightweight monitors to extract critical runtime data.<n>When powered by Gemini 3 Flash Preview, DAIRA establishes a new state-of-the-art (SOTA) performance, achieving a 79.4% resolution rate on the SWE-bench Verified dataset.
  arXiv  Detail & Related papers  (2026-03-23T14:48:54Z)
• AgentTrace: Causal Graph Tracing for Root Cause Analysis in Deployed Multi-Agent Systems [0.0]
  We present AgentTrace, a lightweight causal tracing framework for post-hoc failure diagnosis in deployed multi-agent AI systems.<n>AgentTrace reconstructs causal graphs from execution logs, traces backward from error manifestations, and ranks candidate root causes using interpretable structural and positional signals.<n>Our results suggest that causal tracing provides a practical foundation for improving the reliability and trustworthiness of agentic systems in the wild.
  arXiv  Detail & Related papers  (2026-03-16T00:46:44Z)
• AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents [50.481033105867205]
  We introduce AgentProcessBench, the first benchmark dedicated to evaluating step-level effectiveness in realistic, tool-augmented trajectories.<n>The benchmark comprises 1,000 diverse trajectories and 8,509 human-labeled step annotations with 89.1% inter-annotator agreement.<n>It features a ternary labeling scheme to capture exploration and an error propagation rule to reduce labeling ambiguity.
  arXiv  Detail & Related papers  (2026-03-15T16:13:58Z)
• Understanding by Reconstruction: Reversing the Software Development Process for LLM Pretraining [66.89012795621349]
  Large Language Models (LLMs) often struggle with the deep, long-horizon reasoning required for complex software engineering.<n>We propose a novel paradigm: understanding via reconstruction.<n>We introduce a framework that synthesizes latent agentic trajectories using a multi-agent simulation.
  arXiv  Detail & Related papers  (2026-03-11T09:23:20Z)
• Process-Centric Analysis of Agentic Software Systems [10.976178600911263]
  We introduce Graphectory to encode the temporal and semantic relations in software systems.<n>We analyze 4000 trajectories of two dominant agentic programming models.<n>Our fully automated analyses reveal that agents using richer prompts exhibit more complex Graphectory.
  arXiv  Detail & Related papers  (2025-12-02T04:12:29Z)
• InspectCoder: Dynamic Analysis-Enabled Self Repair through interactive LLM-Debugger Collaboration [71.18377595277018]
  Large Language Models (LLMs) frequently generate buggy code with complex logic errors that are challenging to diagnose.<n>We present InspectCoder, the first agentic program repair system that empowers LLMs to actively conduct dynamic analysis via interactive debugger control.
  arXiv  Detail & Related papers  (2025-10-21T06:26:29Z)
• CoDA: Agentic Systems for Collaborative Data Visualization [57.270599188947294]
  Deep research has revolutionized data analysis, yet data scientists still devote substantial time to manually crafting visualizations.<n>Existing approaches, including simple single- or multi-agent systems, often oversimplify the task.<n>We introduce CoDA, a multi-agent system that employs specialized LLM agents for metadata analysis, task planning, code generation, and self-reflection.
  arXiv  Detail & Related papers  (2025-10-03T17:30:16Z)
• From Trace to Line: LLM Agent for Real-World OSS Vulnerability Localization [14.474705451897691]
  We present T2L-Agent, a project-level, end-to-end framework that plans its own analysis and narrows scope from modules to exact vulnerable lines.<n>We benchmark T2L-ARVO, a diverse, expert-verified 50-case benchmark spanning five crash families and real-world projects.<n>On T2L-ARVO, T2L-Agent achieves up to 58.0% detection and 54.8% line-level localization, substantially outperforming baselines.
  arXiv  Detail & Related papers  (2025-09-30T22:27:18Z)
• Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems [50.29939179830491]
  Failure attribution in LLM multi-agent systems remains underexplored and labor-intensive.<n>We develop and evaluate three automated failure attribution methods, summarizing their corresponding pros and cons.<n>The best method achieves 53.5% accuracy in identifying failure-responsible agents but only 14.2% in pinpointing failure steps.
  arXiv  Detail & Related papers  (2025-04-30T23:09:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.