Can LLMs Really Recover Microservice Failures? A Recovery-Aware Evaluation of Diagnosis-to-Action Reasoning

• URL: http://arxiv.org/abs/2607.04623v1
• Date: Mon, 06 Jul 2026 03:05:17 GMT
• Title: Can LLMs Really Recover Microservice Failures? A Recovery-Aware Evaluation of Diagnosis-to-Action Reasoning
• Abstract summary: This paper presents R2Act, a recovery-action evaluation framework for post-diagnosis incident response.<n>R2Act defines an incident schema, quality gate, action-space representation, recovery-validity metrics, offline evaluator, and live-replay protocol.<n>We instantiate the framework as a benchmark dataset of 302 quality-audited incidents from system.
• Score: 48.22526014304726
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) are increasingly used to interpret operational evidence and assist incident response in cloud-native microservice systems. However, recovery-oriented use cases require more than identifying a root cause. After observing symptoms and diagnosing a fault, an operator or agent must translate the diagnosis into a concrete recovery action, apply it to an admissible target, and verify that service health has been restored. Existing RCA and log-analysis evaluations are well-suited to diagnosis, but they do not characterize this subsequent action decision. This paper presents R2Act, a recovery-action evaluation framework for post-diagnosis incident response. R2Act defines an incident schema, quality gate, action-space representation, recovery-validity metrics, offline evaluator, and live-replay protocol. We instantiate the framework as a benchmark dataset of 302 quality-audited Kubernetes incidents from \system. Each incident provides synchronized multi-modal observations, root-cause labels, an incident-specific action space, and annotated valid and invalid recovery plans. We evaluate heuristic, supervised, RCA-oriented, deep log, and LLM-based methods. The strongest RAG-based LLMs reach 91.4\%--99.7\% root-cause service accuracy, yet their recovery validity remains only 36.8\%--60.3\%. Even when both the root-cause service and fault type are correct, recovery-oriented methods still choose invalid actions for 39.5\%--62.0\% of correctly diagnosed incidents. Overall, this work reveals that many recovery failures arise not from missing diagnostic knowledge, but from the difficulty of translating diagnostic evidence into valid recovery actions and admissible targets. This work provides a reproducible, simplified starting point for research and evaluation.

Related papers

• Beyond Fault Localization: A Trajectory-Level Study of LLM Agents for Microservice Root Cause Analysis [13.865176720288938]
  We use a trajectory-level framework to evaluate root cause analysis (RCA) performance.<n>We find a disconnect between answer correctness and diagnostic quality.<n>We operationalize this taxonomy as DiagGuard, a two-stage defense-in-depth architecture.
  arXiv  Detail & Related papers  (2026-08-21T17:13:45Z)
• Diagnosis Before Recovery: Turning Agent Failures into Selective Self-Correction [22.654396704495934]
  Development-set failures can recover part of the missing diagnostic substrate by deciding which recovery interventions are admissible.<n>We propose DARC, a diagnosis-guided recovery harness that profiles task-family failure modes, prunes mismatched interventions from a shared recovery library, and freezes a verifier-selected success-cost policy for deployment.
  arXiv  Detail & Related papers  (2026-08-12T08:14:45Z)
• Measure, Don't Optimize: Forecasting Recovery in LLM Unlearning [57.920153607557744]
  J-Access is an inference-time audit that measures how often target concepts remain accessible along the model's output pathway.<n>We audit 398 public unlearned models spanning eight unlearning methods.
  arXiv  Detail & Related papers  (2026-08-11T20:16:20Z)
• AgentDebugX: An Open-Source Toolkit for Failure Observability, Attribution, and Recovery in LLM Agents [83.37969790806069]
  LLM agent failures are difficult to debug because the step where an error surfaces is often not the one that caused it.<n>We present Agent DebuggerX, an open-source framework that organizes debug as a closed loop of Detect, Attribute, Recover, and Rerun.<n>At its core, Deep Debugger performs multi-turn root-cause diagnosis through global trajectory understanding.
  arXiv  Detail & Related papers  (2026-07-21T06:21:13Z)
• Can Large Language Models Generate Observability-Aware Code? [51.30941152293301]
  We present a systematic study of observability in agent-generated systems.<n>Our results reveal a consistent gap between diagnostic semantics at the source level and fault signals at runtime.<n>We introduce an observability-oriented skill, which can serve as a guidance to improve both diagnostic semantics and fault-signal exposure.
  arXiv  Detail & Related papers  (2026-07-07T03:20:52Z)
• Where Do Deep-Research Agents Go Wrong? Span-Level Error Localization in Agent Trajectories [51.22051230894794]
  Evaluation based on final answers shows whether an agent succeeds, but not which parts of the trajectory make the answer unreliable.<n>We collect 2,790 real trajectories from two agent frameworks, three backbone models, and three benchmarks, convert raw logs into semantic spans, and annotate harmful error spans through expert review.<n>We propose DRIFT, a claim-centric auditing framework that tracks agent claims, checks their support in trajectory evidence, and marks spans where unsupported or conflicting claims affect the answer path.
  arXiv  Detail & Related papers  (2026-06-01T10:50:26Z)
• Self-Healing Agentic Orchestrators for Reliable Tool-Augmented Large Language Model Systems [0.10210859604701106]
  Self-healing agentic orchestrator treats reliability as a runtime bounded control problem.<n>Self-healing achieves 98.8% task success, compared with 94.5% for retry-only and 93.8% for full replanning.
  arXiv  Detail & Related papers  (2026-05-31T19:27:22Z)
• Debugging the Debuggers: Failure-Anchored Structured Recovery for Software Engineering Agents [26.16025682022058]
  PROBE is a failure-anchored framework for structured recovery in software engineering agents.<n>We evaluate PROBE across three settings: repositorylevel software repair, enterprise workflow recovery, and AIOps service mitigation.
  arXiv  Detail & Related papers  (2026-05-09T06:02:08Z)
• ANNEAL: Adapting LLM Agents via Governed Symbolic Patch Learning [12.44509691077682]
  We introduce ANNEAL, a neuro-symbolic agent that converts recurring failures into governed symbolic edits of a process knowledge graph.<n>Its core mechanism, Failure-Driven Knowledge Acquisition (FDKA), localizes the responsible operator, synthesizes a typed patch through constrained LLM generation, and validates the proposal.<n>Across four domains and 27 multi-seed runs, ANNEAL is the only evaluated system that commits persistent structural repairs.
  arXiv  Detail & Related papers  (2026-05-04T05:24:03Z)
• Doctor-RAG: Failure-Aware Repair for Agentic Retrieval-Augmented Generation [23.961760006898157]
  Agentic Retrieval-Augmented Generation (Agentic RAG) has become a widely adopted paradigm for question answering and complex knowledge reasoning.<n>Existing approaches typically address such failures by either stopping at diagnostic analysis or rerunning the entire retrieval-reasoning pipeline.<n>We propose Doctor-RAG, a unified diagnose-and-repair framework that corrects failures in Agentic RAG through explicit error localization and prefix reuse.
  arXiv  Detail & Related papers  (2026-04-01T13:13:27Z)
• DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems [48.971606069204825]
  DoVer is an intervention-driven debug framework for large language model (LLM)-based multi-agent systems.<n>It augments hypothesis generation with active verification through targeted interventions.<n>DoVer flips 18-28% of failed trials into successes, achieves up to 16% milestone progress, and validates or refutes 30-60% of failure hypotheses.
  arXiv  Detail & Related papers  (2025-12-07T09:23:48Z)
• GALA: Can Graph-Augmented Large Language Model Agentic Workflows Elevate Root Cause Analysis? [9.394057684388027]
  This paper introduces GALA, a novel framework for root cause analysis in microservice systems.<n> evaluated on an open-source benchmark, GALA achieves substantial improvements over state-of-the-art methods.<n>We show that GALA bridges the gap between automated failure diagnosis and practical incident resolution.
  arXiv  Detail & Related papers  (2025-08-17T19:12:05Z)
• Quantifying the Reasoning Abilities of LLMs on Real-world Clinical Cases [48.87360916431396]
  We introduce MedR-Bench, a benchmarking dataset of 1,453 structured patient cases, annotated with reasoning references.<n>We propose a framework encompassing three critical examination recommendation, diagnostic decision-making, and treatment planning, simulating the entire patient care journey.<n>Using this benchmark, we evaluate five state-of-the-art reasoning LLMs, including DeepSeek-R1, OpenAI-o3-mini, and Gemini-2.0-Flash Thinking, etc.
  arXiv  Detail & Related papers  (2025-03-06T18:35:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.