RCAEval: A Benchmark for Root Cause Analysis of Microservice Systems with Telemetry Data

• URL: http://arxiv.org/abs/2412.17015v5
• Date: Mon, 03 Feb 2025 15:35:11 GMT
• Title: RCAEval: A Benchmark for Root Cause Analysis of Microservice Systems with Telemetry Data
• Authors: Luan Pham, Hongyu Zhang, Huong Ha, Flora Salim, Xiuzhen Zhang,
• Abstract summary: Root cause analysis (RCA) for microservice systems has gained significant attention in recent years. There is still no standard benchmark that includes large-scale datasets and supports comprehensive evaluation environments. We introduce RCAEval, an open-source benchmark that provides datasets and an evaluation environment for RCA in microservice systems.
• Score: 13.68949728404533
• License:
• Abstract: Root cause analysis (RCA) for microservice systems has gained significant attention in recent years. However, there is still no standard benchmark that includes large-scale datasets and supports comprehensive evaluation environments. In this paper, we introduce RCAEval, an open-source benchmark that provides datasets and an evaluation environment for RCA in microservice systems. First, we introduce three comprehensive datasets comprising 735 failure cases collected from three microservice systems, covering various fault types observed in real-world failures. Second, we present a comprehensive evaluation framework that includes fifteen reproducible baselines covering a wide range of RCA approaches, with the ability to evaluate both coarse-grained and fine-grained RCA. We hope that this ready-to-use benchmark will enable researchers and practitioners to conduct extensive analysis and pave the way for robust new solutions for RCA of microservice systems.

Related papers

• Network Centrality as a New Perspective on Microservice Architecture [48.55946052680251]
  The adoption of Microservice Architecture has led to the identification of various patterns and anti-patterns, such as Nano/Mega/Hub services. This study investigates whether centrality metrics (CMs) can provide new insights into MSA quality and facilitate the detection of architectural anti-patterns.
  arXiv  Detail & Related papers  (2025-01-23T10:13:57Z)
• RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework [69.4501863547618]
  This paper introduces RAGEval, a framework designed to assess RAG systems across diverse scenarios. With a focus on factual accuracy, we propose three novel metrics Completeness, Hallucination, and Irrelevance. Experimental results show that RAGEval outperforms zero-shot and one-shot methods in terms of clarity, safety, conformity, and richness of generated samples.
  arXiv  Detail & Related papers  (2024-08-02T13:35:11Z)
• CHASE: A Causal Heterogeneous Graph based Framework for Root Cause Analysis in Multimodal Microservice Systems [22.00860661894853]
  We propose a Causal Heterogeneous grAph baSed framEwork for root cause analysis, namely CHASE, for microservice systems with multimodal data. CHASE learns from the constructed hypergraph with hyperedges representing the flow of causality and performs root cause localization.
  arXiv  Detail & Related papers  (2024-06-28T07:46:51Z)
• LEMMA-RCA: A Large Multi-modal Multi-domain Dataset for Root Cause Analysis [32.816594249593955]
  Root cause analysis (RCA) is crucial for enhancing the reliability and performance of complex systems. We introduce LEMMA-RCA, a large dataset designed for diverse RCA tasks across multiple domains and modalities. We evaluate the quality of LEMMA-RCA by testing the performance of eight baseline methods on this dataset.
  arXiv  Detail & Related papers  (2024-06-08T07:00:31Z)
• Decoding the MITRE Engenuity ATT&CK Enterprise Evaluation: An Analysis of EDR Performance in Real-World Environments [4.554190270271114]
  In light of the growing significance of endpoint detection and response (EDR) systems, many cybersecurity providers have developed their own solutions. It's crucial for users to assess the capabilities of these detection engines to make informed decisions about which products to choose. In this paper, we thoroughly analyzed MITRE evaluation results to gain further insights into real-world systems under test.
  arXiv  Detail & Related papers  (2024-01-29T04:19:56Z)
• A Survey on Open-Set Image Recognition [18.474539379698538]
  Open-set image recognition (OSR) aims to both classify known-class samples and identify unknown-class samples in the testing set. We introduce a new taxonomy, under which we comprehensively review the existing DNN-based OSR methods. We compare the performances of some typical and state-of-the-art OSR methods on both coarse-grained datasets and fine-grained datasets.
  arXiv  Detail & Related papers  (2023-12-25T00:30:23Z)
• TraceDiag: Adaptive, Interpretable, and Efficient Root Cause Analysis on Large-Scale Microservice Systems [44.53009495726297]
  Root Cause Analysis (RCA) is becoming increasingly crucial for ensuring the reliability of microservice systems. This paper proposes TraceDiag, an end-to-end RCA framework that addresses the challenges for large-scale microservice systems.
  arXiv  Detail & Related papers  (2023-10-28T15:49:00Z)
• A Survey on Interpretable Cross-modal Reasoning [64.37362731950843]
  Cross-modal reasoning (CMR) has emerged as a pivotal area with applications spanning from multimedia analysis to healthcare diagnostics. This survey delves into the realm of interpretable cross-modal reasoning (I-CMR) This survey presents a comprehensive overview of the typical methods with a three-level taxonomy for I-CMR.
  arXiv  Detail & Related papers  (2023-09-05T05:06:48Z)
• Automatic Root Cause Analysis via Large Language Models for Cloud Incidents [51.94361026233668]
  We introduce RCACopilot, an on-call system empowered by a large language model for automating root cause analysis of cloud incidents. RCACopilot matches incoming incidents to corresponding incident handlers based on their alert types, aggregates the critical runtime diagnostic information, predicts the incident's root cause category, and provides an explanatory narrative. We evaluate RCACopilot using a real-world dataset consisting of a year's worth of incidents from Microsoft.
  arXiv  Detail & Related papers  (2023-05-25T06:44:50Z)
• Wider or Deeper Neural Network Architecture for Acoustic Scene Classification with Mismatched Recording Devices [59.86658316440461]
  We present a robust and low complexity system for Acoustic Scene Classification (ASC) We first construct an ASC baseline system in which a novel inception-residual-based network architecture is proposed to deal with the mismatched recording device issue. To further improve the performance but still satisfy the low complexity model, we apply two techniques: ensemble of multiple spectrograms and channel reduction.
  arXiv  Detail & Related papers  (2022-03-23T10:27:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.