FaultProfIT: Hierarchical Fault Profiling of Incident Tickets in
Large-scale Cloud Systems
- URL: http://arxiv.org/abs/2402.17583v1
- Date: Tue, 27 Feb 2024 15:14:19 GMT
- Title: FaultProfIT: Hierarchical Fault Profiling of Incident Tickets in
Large-scale Cloud Systems
- Authors: Junjie Huang, Jinyang Liu, Zhuangbin Chen, Zhihan Jiang, Yichen LI,
Jiazhen Gu, Cong Feng, Zengyin Yang, Yongqiang Yang, Michael R. Lyu
- Abstract summary: We propose an automated approach, named FaultProfIT, for Fault pattern Profiling of Incident Tickets.
It leverages hierarchy-guided contrastive learning to train a hierarchy-aware incident encoder and predicts fault patterns with enhanced incident representations.
To date, FaultProfIT has analyzed 10,000+ incidents from 30+ cloud services, successfully revealing several fault trends that have informed system improvements.
- Score: 35.310727641258715
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: Postmortem analysis is essential in the management of incidents within cloud
systems, which provides valuable insights to improve system's reliability and
robustness. At CloudA, fault pattern profiling is performed during the
postmortem phase, which involves the classification of incidents' faults into
unique categories, referred to as fault pattern. By aggregating and analyzing
these fault patterns, engineers can discern common faults, vulnerable
components and emerging fault trends. However, this process is currently
conducted by manual labeling, which has inherent drawbacks. On the one hand,
the sheer volume of incidents means only the most severe ones are analyzed,
causing a skewed overview of fault patterns. On the other hand, the complexity
of the task demands extensive domain knowledge, which leads to errors and
inconsistencies. To address these limitations, we propose an automated
approach, named FaultProfIT, for Fault pattern Profiling of Incident Tickets.
It leverages hierarchy-guided contrastive learning to train a hierarchy-aware
incident encoder and predicts fault patterns with enhanced incident
representations. We evaluate FaultProfIT using the production incidents from
CloudA. The results demonstrate that FaultProfIT outperforms state-of-the-art
methods. Our ablation study and analysis also verify the effectiveness of
hierarchy-guided contrastive learning. Additionally, we have deployed
FaultProfIT at CloudA for six months. To date, FaultProfIT has analyzed 10,000+
incidents from 30+ cloud services, successfully revealing several fault trends
that have informed system improvements.
Related papers
- Reshaping the Online Data Buffering and Organizing Mechanism for Continual Test-Time Adaptation [49.53202761595912]
Continual Test-Time Adaptation involves adapting a pre-trained source model to continually changing unsupervised target domains.
We analyze the challenges of this task: online environment, unsupervised nature, and the risks of error accumulation and catastrophic forgetting.
We propose an uncertainty-aware buffering approach to identify and aggregate significant samples with high certainty from the unsupervised, single-pass data stream.
arXiv Detail & Related papers (2024-07-12T15:48:40Z) - FaultGuard: A Generative Approach to Resilient Fault Prediction in Smart Electrical Grids [53.2306792009435]
FaultGuard is the first framework for fault type and zone classification resilient to adversarial attacks.
We propose a low-complexity fault prediction model and an online adversarial training technique to enhance robustness.
Our model outclasses the state-of-the-art for resilient fault prediction benchmarking, with an accuracy of up to 0.958.
arXiv Detail & Related papers (2024-03-26T08:51:23Z) - Root Causing Prediction Anomalies Using Explainable AI [3.970146574042422]
We present a novel application of explainable AI (XAI) for root-causing performance degradation in machine learning models.
A single feature corruption can cause cascading feature, label and concept drifts.
We have successfully applied this technique to improve the reliability of models used in personalized advertising.
arXiv Detail & Related papers (2024-03-04T19:38:50Z) - KGroot: Enhancing Root Cause Analysis through Knowledge Graphs and Graph
Convolutional Neural Networks [14.336830860792707]
KGroot uses event knowledge and the correlation between events to perform root cause reasoning.
Experiments demonstrate KGroot can locate the root cause with accuracy of 93.5% top 3 potential causes in second-level.
arXiv Detail & Related papers (2024-02-11T10:30:38Z) - Progressing from Anomaly Detection to Automated Log Labeling and
Pioneering Root Cause Analysis [53.24804865821692]
This study introduces a taxonomy for log anomalies and explores automated data labeling to mitigate labeling challenges.
The study envisions a future where root cause analysis follows anomaly detection, unraveling the underlying triggers of anomalies.
arXiv Detail & Related papers (2023-12-22T15:04:20Z) - STEAM & MoSAFE: SOTIF Error-and-Failure Model & Analysis for AI-Enabled
Driving Automation [4.820785104084241]
This paper defines the SOTIF Temporal Error and Failure Model (STEAM) as a refinement of the SOTIF cause-and-effect model.
Second, this paper proposes the Model-based SOTIF Analysis of Failures and Errors (MoSAFE) method, which allows instantiating STEAM based on system-design models.
arXiv Detail & Related papers (2023-12-15T06:34:35Z) - Causal Disentanglement Hidden Markov Model for Fault Diagnosis [55.90917958154425]
We propose a Causal Disentanglement Hidden Markov model (CDHM) to learn the causality in the bearing fault mechanism.
Specifically, we make full use of the time-series data and progressively disentangle the vibration signal into fault-relevant and fault-irrelevant factors.
To expand the scope of the application, we adopt unsupervised domain adaptation to transfer the learned disentangled representations to other working environments.
arXiv Detail & Related papers (2023-08-06T05:58:45Z) - Repairing Neural Networks by Leaving the Right Past Behind [23.78437548836594]
Prediction failures of machine learning models often arise from deficiencies in training data.
This work develops a generic framework for both identifying training examples that have given rise to the target failure, and fixing the model through erasing information about them.
arXiv Detail & Related papers (2022-07-11T12:07:39Z) - Fast and Accurate Error Simulation for CNNs against Soft Errors [64.54260986994163]
We present a framework for the reliability analysis of Conal Neural Networks (CNNs) via an error simulation engine.
These error models are defined based on the corruption patterns of the output of the CNN operators induced by faults.
We show that our methodology achieves about 99% accuracy of the fault effects w.r.t. SASSIFI, and a speedup ranging from 44x up to 63x w.r.t.FI, that only implements a limited set of error models.
arXiv Detail & Related papers (2022-06-04T19:45:02Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.