Beyond Fault Localization: A Trajectory-Level Study of LLM Agents for Microservice Root Cause Analysis

• URL: http://arxiv.org/abs/2608.21310v1
• Date: Fri, 21 Aug 2026 17:13:45 GMT
• Title: Beyond Fault Localization: A Trajectory-Level Study of LLM Agents for Microservice Root Cause Analysis
• Abstract summary: We use a trajectory-level framework to evaluate root cause analysis (RCA) performance.<n>We find a disconnect between answer correctness and diagnostic quality.<n>We operationalize this taxonomy as DiagGuard, a two-stage defense-in-depth architecture.
• Score: 13.865176720288938
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing evaluations of automated root cause analysis (RCA) for microservices assess diagnostic performance mainly by endpoint correctness: whether a method localizes the responsible service. This criterion enables comparison but does not reveal the evidentiary basis of a diagnosis or the fault-propagation route connecting the source to observed symptoms, both of which an on-call site reliability engineer needs to judge whether action is warranted. We therefore treat RCA as an observable diagnostic process. Our trajectory-level framework evaluates agent executions against manually curated service-level fault-propagation paths. Applied to a public microservice RCA benchmark, it analyzes 3,500 diagnostic trajectories, characterizing where agents investigate and how they use retrieved telemetry. We find a disconnect between answer correctness and diagnostic quality: an agent may localize the fault source yet fail to reconstruct its propagation. Successful investigations stay on the fault-impact surface, act on retrieved evidence, and broaden their query repertoire as the search deepens. Failures arise when decisive evidence is omitted, retrieved evidence is misinterpreted, or unsupported inference substitutes for missing evidence. We operationalize this taxonomy as DiagGuard, a two-stage defense-in-depth architecture in which grounding surveys available observations before localization and verification audits the diagnosis against them. In an independent setting with a different model, benchmark, and service topology, DiagGuard raises Acc@1 from 43.5% to 52.5%. These results show that trajectory-level evaluation exposes limitations hidden by final-answer metrics and provides actionable guidance for improving automated RCA.

Related papers

• Can Large Language Models Generate Observability-Aware Code? [51.30941152293301]
  We present a systematic study of observability in agent-generated systems.<n>Our results reveal a consistent gap between diagnostic semantics at the source level and fault signals at runtime.<n>We introduce an observability-oriented skill, which can serve as a guidance to improve both diagnostic semantics and fault-signal exposure.
  arXiv  Detail & Related papers  (2026-07-07T03:20:52Z)
• Can LLMs Really Recover Microservice Failures? A Recovery-Aware Evaluation of Diagnosis-to-Action Reasoning [48.22526014304726]
  This paper presents R2Act, a recovery-action evaluation framework for post-diagnosis incident response.<n>R2Act defines an incident schema, quality gate, action-space representation, recovery-validity metrics, offline evaluator, and live-replay protocol.<n>We instantiate the framework as a benchmark dataset of 302 quality-audited incidents from system.
  arXiv  Detail & Related papers  (2026-07-06T03:05:17Z)
• Anomaly Detection and Root Cause Analysis for Microservice Systems [0.0]
  Microservice systems are widely used to build cloud applications, yet their complexity makes failures inevitable.<n>This thesis advances automated anomaly detection and root cause analysis for microservice failures.
  arXiv  Detail & Related papers  (2026-06-08T00:50:43Z)
• Where Do Deep-Research Agents Go Wrong? Span-Level Error Localization in Agent Trajectories [51.22051230894794]
  Evaluation based on final answers shows whether an agent succeeds, but not which parts of the trajectory make the answer unreliable.<n>We collect 2,790 real trajectories from two agent frameworks, three backbone models, and three benchmarks, convert raw logs into semantic spans, and annotate harmful error spans through expert review.<n>We propose DRIFT, a claim-centric auditing framework that tracks agent claims, checks their support in trajectory evidence, and marks spans where unsupported or conflicting claims affect the answer path.
  arXiv  Detail & Related papers  (2026-06-01T10:50:26Z)
• Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents? [61.49434544687523]
  We introduce REFLECT, a meta-evaluation benchmark targeting fine-grained failure detection in agentic environments.<n> REFLECT defines a detailed taxonomy of process- and outcome-level failure modes, instantiated by performing controlled and localized interventions.<n>Our experiments show that even the best-performing models achieve overall accuracies below 55% across reasoning, tool-use, and report-quality failures.
  arXiv  Detail & Related papers  (2026-05-18T23:55:08Z)
• Debugging the Debuggers: Failure-Anchored Structured Recovery for Software Engineering Agents [26.16025682022058]
  PROBE is a failure-anchored framework for structured recovery in software engineering agents.<n>We evaluate PROBE across three settings: repositorylevel software repair, enterprise workflow recovery, and AIOps service mitigation.
  arXiv  Detail & Related papers  (2026-05-09T06:02:08Z)
• SAAG: Structured Agent Assessment and Grounding [13.62148061055744]
  We propose a cascaded diagnostic framework that decomposes agent-calling evaluation into three sequential stages.<n>We evaluate this framework on a benchmark derived from Glaive's function-calling dataset across registry sizes of 5, 10, and 15 agents.
  arXiv  Detail & Related papers  (2026-04-30T19:33:58Z)
• Automated Self-Testing as a Quality Gate: Evidence-Driven Release Management for LLM Applications [51.56484100374058]
  We present an automated self-testing framework that introduces quality gates with evidence-based release decisions.<n>We evaluate the framework through a longitudinal case study of an internally deployed multi-agent conversational AI system.
  arXiv  Detail & Related papers  (2026-03-13T20:44:15Z)
• Guideline-Grounded Evidence Accumulation for High-Stakes Agent Verification [60.18369393468405]
  Existing verifiers usually underperform owing to a lack of domain knowledge and limited calibration.<n>GLEAN compiles expert-curated protocols into trajectory-informed, well-calibrated correctness signals.<n>We empirically validate GLEAN with agentic clinical diagnosis across three diseases from the MIMIC-IV dataset.
  arXiv  Detail & Related papers  (2026-03-03T09:36:43Z)
• GALA: Can Graph-Augmented Large Language Model Agentic Workflows Elevate Root Cause Analysis? [9.394057684388027]
  This paper introduces GALA, a novel framework for root cause analysis in microservice systems.<n> evaluated on an open-source benchmark, GALA achieves substantial improvements over state-of-the-art methods.<n>We show that GALA bridges the gap between automated failure diagnosis and practical incident resolution.
  arXiv  Detail & Related papers  (2025-08-17T19:12:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.