MTAD: Tools and Benchmarks for Multivariate Time Series Anomaly Detection

• URL: http://arxiv.org/abs/2401.06175v1
• Date: Wed, 10 Jan 2024 06:50:25 GMT
• Title: MTAD: Tools and Benchmarks for Multivariate Time Series Anomaly Detection
• Authors: Jinyang Liu, Wenwei Gu, Zhuangbin Chen, Yichen Li, Yuxin Su, Michael R. Lyu
• Abstract summary: We provide a comprehensive review and evaluation of twelve state-of-the-art anomaly detection methods. We propose a novel metric called salience to better understand the characteristics of different anomaly detectors. We report the benchmark results in terms of accuracy, salience, efficiency, and delay, which are of practical importance for industrial deployment.
• Score: 34.81779490744863
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Key Performance Indicators (KPIs) are essential time-series metrics for ensuring the reliability and stability of many software systems. They faithfully record runtime states to facilitate the understanding of anomalous system behaviors and provide informative clues for engineers to pinpoint the root causes. The unprecedented scale and complexity of modern software systems, however, make the volume of KPIs explode. Consequently, many traditional methods of KPI anomaly detection become impractical, which serves as a catalyst for the fast development of machine learning-based solutions in both academia and industry. However, there is currently a lack of rigorous comparison among these KPI anomaly detection methods, and re-implementation demands a non-trivial effort. Moreover, we observe that different works adopt independent evaluation processes with different metrics. Some of them may not fully reveal the capability of a model and some are creating an illusion of progress. To better understand the characteristics of different KPI anomaly detectors and address the evaluation issue, in this paper, we provide a comprehensive review and evaluation of twelve state-of-the-art methods, and propose a novel metric called salience. Particularly, the selected methods include five traditional machine learning-based methods and seven deep learning-based methods. These methods are evaluated with five multivariate KPI datasets that are publicly available. A unified toolkit with easy-to-use interfaces is also released. We report the benchmark results in terms of accuracy, salience, efficiency, and delay, which are of practical importance for industrial deployment. We believe our work can contribute as a basis for future academic research and industrial application.

Related papers

• A Comprehensive Library for Benchmarking Multi-class Visual Anomaly Detection [52.228708947607636]
  This paper introduces a comprehensive visual anomaly detection benchmark, ADer, which is a modular framework for new methods. The benchmark includes multiple datasets from industrial and medical domains, implementing fifteen state-of-the-art methods and nine comprehensive metrics. We objectively reveal the strengths and weaknesses of different methods and provide insights into the challenges and future directions of multi-class visual anomaly detection.
  arXiv  Detail & Related papers  (2024-06-05T13:40:07Z)
• Malicious code detection in android: the role of sequence characteristics and disassembling methods [0.0]
  We investigate and emphasize the factors that may affect the accuracy values of the models managed by researchers. Our findings exhibit that the disassembly method and different input representations affect the model results.
  arXiv  Detail & Related papers  (2023-12-02T11:55:05Z)
• Interactive System-wise Anomaly Detection [66.3766756452743]
  Anomaly detection plays a fundamental role in various applications. It is challenging for existing methods to handle the scenarios where the instances are systems whose characteristics are not readily observed as data. We develop an end-to-end approach which includes an encoder-decoder module that learns system embeddings.
  arXiv  Detail & Related papers  (2023-04-21T02:20:24Z)
• SECOE: Alleviating Sensors Failure in Machine Learning-Coupled IoT Systems [0.0]
  This paper proposes SECOE, a proactive approach for alleviating potentially simultaneous sensor failures. SECOE includes a novel technique to minimize the number of models in the ensemble by harnessing the correlations among sensors. Experiments reveal that SECOE effectively preserves prediction accuracy in the presence of sensor failures.
  arXiv  Detail & Related papers  (2022-10-05T10:58:39Z)
• What are the best systems? New perspectives on NLP Benchmarking [10.27421161397197]
  We propose a new procedure to rank systems based on their performance across different tasks. Motivated by the social choice theory, the final system ordering is obtained through aggregating the rankings induced by each task. We show that our method yields different conclusions on state-of-the-art systems than the mean-aggregation procedure.
  arXiv  Detail & Related papers  (2022-02-08T11:44:20Z)
• DeepTimeAnomalyViz: A Tool for Visualizing and Post-processing Deep Learning Anomaly Detection Results for Industrial Time-Series [88.12892448747291]
  We introduce the DeTAVIZ interface, which is a web browser based visualization tool for quick exploration and assessment of feasibility of DL based anomaly detection in a given problem. DeTAVIZ allows the user to easily and quickly iterate through multiple post processing options and compare different models, and allows for manual optimisation towards a chosen metric.
  arXiv  Detail & Related papers  (2021-09-21T10:38:26Z)
• An Extensible Benchmark Suite for Learning to Simulate Physical Systems [60.249111272844374]
  We introduce a set of benchmark problems to take a step towards unified benchmarks and evaluation protocols. We propose four representative physical systems, as well as a collection of both widely used classical time-based and representative data-driven methods.
  arXiv  Detail & Related papers  (2021-08-09T17:39:09Z)
• Experience Report: Deep Learning-based System Log Analysis for Anomaly Detection [30.52620190783608]
  We provide a review and evaluation on five popular models used by six state-of-the-art anomaly detectors. Four of the selected methods are unsupervised and the remaining two are supervised. We believe our work can serve as a basis in this field and contribute to the future academic researches and industrial applications.
  arXiv  Detail & Related papers  (2021-07-13T08:10:47Z)
• ALT-MAS: A Data-Efficient Framework for Active Testing of Machine Learning Algorithms [58.684954492439424]
  We propose a novel framework to efficiently test a machine learning model using only a small amount of labeled test data. The idea is to estimate the metrics of interest for a model-under-test using Bayesian neural network (BNN)
  arXiv  Detail & Related papers  (2021-04-11T12:14:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.