A Multi-Dataset Benchmark for Evaluating LLM Agents in Microservice Failure Diagnosis

• URL: http://arxiv.org/abs/2606.29193v1
• Date: Sun, 28 Jun 2026 04:38:05 GMT
• Title: A Multi-Dataset Benchmark for Evaluating LLM Agents in Microservice Failure Diagnosis
• Abstract summary: LLM-based agents are reshaping microservice operations into AgentOps, where benchmarks are key to evaluating failure diagnosis over multimodal observability data.<n>We introduce two large-scale datasets (AIOps2025 and RCA100) under a reasoning-process evaluation paradigm.<n>Together, the two datasets comprise over 500 expert-labeled failure cases across two representative microservice systems.
• Score: 8.941814067287673
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: LLM-based agents are reshaping microservice operations into AgentOps, where benchmarks are key to evaluating failure diagnosis over multimodal observability data. However, existing benchmarks remain largely outcome-oriented: they score only the final answer and fail to assess the systematic reasoning process in failure diagnosis. We address this gap by introducing two large-scale datasets (AIOps2025 and RCA100) under a reasoning-process evaluation paradigm that assesses agentic diagnostic capability along three dimensions: Localization (where the fault occurs), Identification (what type of fault it is), and Reason (whether the reasoning trace is grounded in relevant evidence). Together, the two datasets comprise over 500 expert-labeled failure cases across two representative microservice systems (HipsterShop and the OpenTelemetry Demo Store). They cover diverse fault scenarios across resource, network, runtime, middleware/database, and application-logic categories and provide fine-grained causal evidence to support agent learning and reasoning-process evaluation. Beyond scale and coverage, the datasets have been carefully labelled by domain experts and validated through large-scale competitions, supporting more than 6,000 participating teams. This makes them not only expert-labeled diagnostic datasets, but also competition-validated benchmarks for evaluating agentic failure diagnosis in real-world microservice environments. Datasets are available at https://www.aiops.cn/gitlab/aiops-live-benchmark/agenticopseval.

Related papers

• NovGauge: A Fine-Grained Benchmark for Diagnosing LLMs' Capability in Paper Novelty Assessment [54.4265627247104]
  Large language models (LLMs) are increasingly used in peer review at major AI conferences.<n>Existing benchmarks assess novelty as a single holistic score.<n>We present NovGauge, a human-anchored benchmark for fine-grained novelty assessment diagnosis.
  arXiv  Detail & Related papers  (2026-09-10T08:34:56Z)
• Beyond Fault Localization: A Trajectory-Level Study of LLM Agents for Microservice Root Cause Analysis [13.865176720288938]
  We use a trajectory-level framework to evaluate root cause analysis (RCA) performance.<n>We find a disconnect between answer correctness and diagnostic quality.<n>We operationalize this taxonomy as DiagGuard, a two-stage defense-in-depth architecture.
  arXiv  Detail & Related papers  (2026-08-21T17:13:45Z)
• Anomaly Detection and Root Cause Analysis for Microservice Systems [0.0]
  Microservice systems are widely used to build cloud applications, yet their complexity makes failures inevitable.<n>This thesis advances automated anomaly detection and root cause analysis for microservice failures.
  arXiv  Detail & Related papers  (2026-06-08T00:50:43Z)
• DataClaw: A Process-Oriented Agent Benchmark for Exploratory Real-World Data Analysis [76.98578575566184]
  DataClaw is a process-oriented benchmark for exploratory real-world data analysis.<n>It contains approximately 2.06 million real-world records across enterprise, industry and policy domains.<n>DataClaw measures how far an agent progresses and where its reasoning breaks down.
  arXiv  Detail & Related papers  (2026-05-04T11:57:09Z)
• LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation [75.05397479715576]
  Large language models (LLMs) and agents have shown promising progress, but their true capabilities and failure modes remain unclear.<n>We present the first systematic and contamination-aware study of LLM- and agent-based formal specification generation for C programs.
  arXiv  Detail & Related papers  (2026-05-02T11:31:33Z)
• HippoCamp: Benchmarking Contextual Agents on Personal Computers [71.97629614361549]
  HippoCamp is a new benchmark designed to evaluate agents' capabilities on multimodal file management.<n>Our benchmark instantiates device-scale file systems over real-world profiles spanning diverse modalities, comprising 42.4 GB of data across over 2K real-world files.
  arXiv  Detail & Related papers  (2026-04-01T17:58:33Z)
• PJB: A Reasoning-Aware Benchmark for Person-Job Retrieval [21.108322341202815]
  Person-job matching requires systems to verify explicit constraints and perform skill-transfer inference and job-competency reasoning.<n>Existing benchmarks provide no systematic diagnostic support for this task.<n>We introduce PJB, a reasoning-aware retrieval evaluation dataset.
  arXiv  Detail & Related papers  (2026-03-18T06:08:06Z)
• Automated Self-Testing as a Quality Gate: Evidence-Driven Release Management for LLM Applications [51.56484100374058]
  We present an automated self-testing framework that introduces quality gates with evidence-based release decisions.<n>We evaluate the framework through a longitudinal case study of an internally deployed multi-agent conversational AI system.
  arXiv  Detail & Related papers  (2026-03-13T20:44:15Z)
• AnoMod: A Dataset for Anomaly Detection and Root Cause Analysis in Microservice Systems [18.34761164400137]
  We introduce a new multimodal anomaly dataset built on two open-source microservice systems: SocialNetwork and TrainTicket.<n>For each scenario, we collect five modalities (Mod): logs, metrics, distributed traces, API responses, and code coverage reports.<n>This dataset enables evaluation of cross-modal anomaly detection and fusion/ablation strategies.
  arXiv  Detail & Related papers  (2026-01-30T12:03:51Z)
• DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems [48.971606069204825]
  DoVer is an intervention-driven debug framework for large language model (LLM)-based multi-agent systems.<n>It augments hypothesis generation with active verification through targeted interventions.<n>DoVer flips 18-28% of failed trials into successes, achieves up to 16% milestone progress, and validates or refutes 30-60% of failure hypotheses.
  arXiv  Detail & Related papers  (2025-12-07T09:23:48Z)
• Automating Dataset Updates Towards Reliable and Timely Evaluation of Large Language Models [81.27391252152199]
  Large language models (LLMs) have achieved impressive performance across various natural language benchmarks. We propose to automate dataset updating and provide systematic analysis regarding its effectiveness. There are two updating strategies: 1) mimicking strategy to generate similar samples based on original data, and 2) extending strategy that further expands existing samples.
  arXiv  Detail & Related papers  (2024-02-19T07:15:59Z)
• Robust Failure Diagnosis of Microservice System through Multimodal Data [14.720995687799668]
  We propose DiagFusion, a robust failure diagnosis approach that uses multimodal data. Our evaluations show that DiagFusion outperforms existing methods in terms of root cause instance localization and failure type determination.
  arXiv  Detail & Related papers  (2023-02-21T08:28:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.