DeCorus: Hierarchical Multivariate Anomaly Detection at Cloud-Scale

• URL: http://arxiv.org/abs/2202.06892v1
• Date: Mon, 14 Feb 2022 17:33:00 GMT
• Title: DeCorus: Hierarchical Multivariate Anomaly Detection at Cloud-Scale
• Authors: Bruno Wassermann, David Ohana, Ronen Schaffer, Robert Shahla, Elliot K. Kolodner, Eran Raichstein, Michal Malka
• Abstract summary: We describe the implementation of DeCorus an online log anomaly detection tool for network device syslog messages deployed at a cloud service provider. We use real-world data sets that consist of $1.5$ billion network device syslog messages and hundreds of incident tickets to characterize the performance of DeCorus.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multivariate anomaly detection can be used to identify outages within large volumes of telemetry data for computing systems. However, developing an efficient anomaly detector that can provide users with relevant information is a challenging problem. We introduce our approach to hierarchical multivariate anomaly detection called DeCorus, a statistical multivariate anomaly detector which achieves linear complexity. It extends standard statistical techniques to improve their ability to find relevant anomalies within noisy signals and makes use of types of domain knowledge that system operators commonly possess to compute system-level anomaly scores. We describe the implementation of DeCorus an online log anomaly detection tool for network device syslog messages deployed at a cloud service provider. We use real-world data sets that consist of $1.5$ billion network device syslog messages and hundreds of incident tickets to characterize the performance of DeCorus and compare its ability to detect incidents with five alternative anomaly detectors. While DeCorus outperforms the other anomaly detectors, all of them are challenged by our data set. We share how DeCorus provides value in the field and how we plan to improve its incident detection accuracy.

Related papers

• Scalable Temporal Anomaly Causality Discovery in Large Systems: Achieving Computational Efficiency with Binary Anomaly Flag Data [0.0]
  This study proposes an anomaly causal discovery approach (AnomalyCD) It addresses the accuracy and computational challenges of generating causal graphs from binary flag data sets. Results demonstrate the considerable reduction of the computation overhead and moderate enhancement of the accuracy of temporal causal discovery.
  arXiv  Detail & Related papers  (2024-12-16T14:11:28Z)
• PULL: Reactive Log Anomaly Detection Based On Iterative PU Learning [58.85063149619348]
  We propose PULL, an iterative log analysis method for reactive anomaly detection based on estimated failure time windows. Our evaluation shows that PULL consistently outperforms ten benchmark baselines across three different datasets.
  arXiv  Detail & Related papers  (2023-01-25T16:34:43Z)
• Autoencoder based Anomaly Detection and Explained Fault Localization in Industrial Cooling Systems [0.32956425475258266]
  We present an autoencoder based end-to-end workflow for anomaly detection in large industrial cooling systems. We identify system failures using a threshold on the total reconstruction error. For fault localization, we compute the individual reconstruction error.
  arXiv  Detail & Related papers  (2022-10-14T09:28:49Z)
• A Robust and Explainable Data-Driven Anomaly Detection Approach For Power Electronics [56.86150790999639]
  We present two anomaly detection and classification approaches, namely the Matrix Profile algorithm and anomaly transformer. The Matrix Profile algorithm is shown to be well suited as a generalizable approach for detecting real-time anomalies in streaming time-series data. A series of custom filters is created and added to the detector to tune its sensitivity, recall, and detection accuracy.
  arXiv  Detail & Related papers  (2022-09-23T06:09:35Z)
• Deep Federated Anomaly Detection for Multivariate Time Series Data [93.08977495974978]
  We present a Federated Exemplar-based Deep Neural Network (Fed-ExDNN) to conduct anomaly detection for multivariate time series data on different edge devices. We show that ExDNN and Fed-ExDNN can outperform state-of-the-art anomaly detection algorithms and federated learning techniques.
  arXiv  Detail & Related papers  (2022-05-09T05:06:58Z)
• Sintel: A Machine Learning Framework to Extract Insights from Signals [13.04826679898367]
  We introduce Sintel, a machine learning framework for end-to-end time series tasks such as anomaly detection. Sintel logs the entire anomaly detection journey, providing detailed documentation of anomalies over time. It enables users to analyze signals, compare methods, and investigate anomalies through an interactive visualization tool.
  arXiv  Detail & Related papers  (2022-04-19T19:38:27Z)
• DAE : Discriminatory Auto-Encoder for multivariate time-series anomaly detection in air transportation [68.8204255655161]
  We propose a novel anomaly detection model called Discriminatory Auto-Encoder (DAE) It uses the baseline of a regular LSTM-based auto-encoder but with several decoders, each getting data of a specific flight phase. Results show that the DAE achieves better results in both accuracy and speed of detection.
  arXiv  Detail & Related papers  (2021-09-08T14:07:55Z)
• From One to Many: A Deep Learning Coincident Gravitational-Wave Search [58.720142291102135]
  We construct a two-detector search for gravitational waves from binary black hole mergers using neural networks trained on non-spinning binary black hole data from a single detector. We find that none of these simple two-detector networks are capable of improving the sensitivity over applying networks individually to the data from the detectors.
  arXiv  Detail & Related papers  (2021-08-24T13:25:02Z)
• Graph Neural Network-Based Anomaly Detection in Multivariate Time Series [17.414474298706416]
  We develop a new way to detect anomalies in high-dimensional time series data. Our approach combines a structure learning approach with graph neural networks. We show that our method detects anomalies more accurately than baseline approaches.
  arXiv  Detail & Related papers  (2021-06-13T09:07:30Z)
• TadGAN: Time Series Anomaly Detection Using Generative Adversarial Networks [73.01104041298031]
  TadGAN is an unsupervised anomaly detection approach built on Generative Adversarial Networks (GANs) To capture the temporal correlations of time series, we use LSTM Recurrent Neural Networks as base models for Generators and Critics. To demonstrate the performance and generalizability of our approach, we test several anomaly scoring techniques and report the best-suited one.
  arXiv  Detail & Related papers  (2020-09-16T15:52:04Z)
• Detection of Thin Boundaries between Different Types of Anomalies in Outlier Detection using Enhanced Neural Networks [3.9715120586766584]
  We introduce new types of anomalies called Collective Normal Anomaly and Collective Point Anomaly. Basic domain-independent methods are introduced to detect these defined anomalies in both unsupervised and supervised datasets. The Multi-Layer Perceptron Neural Network is enhanced using the Genetic Algorithm to detect newly defined anomalies with higher precision.
  arXiv  Detail & Related papers  (2020-01-24T21:52:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.