TimeSeriesBench: An Industrial-Grade Benchmark for Time Series Anomaly
Detection Models
- URL: http://arxiv.org/abs/2402.10802v2
- Date: Mon, 26 Feb 2024 14:13:52 GMT
- Title: TimeSeriesBench: An Industrial-Grade Benchmark for Time Series Anomaly
Detection Models
- Authors: Haotian Si, Changhua Pei, Hang Cui, Jingwen Yang, Yongqian Sun,
Shenglin Zhang, Jingjing Li, Haiming Zhang, Jing Han, Dan Pei, Jianhui Li,
Gaogang Xie
- Abstract summary: Time series anomaly detection (TSAD) has attracted considerable scholarly and industrial interest.
However, existing algorithms exhibit a gap in terms of training paradigm, online detection paradigm, and evaluation criteria.
We propose TimeSeriesBench, an industrial-grade benchmark that we continuously maintain as a leaderboard.
- Score: 22.23993481906648
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Driven by the proliferation of real-world application scenarios and scales,
time series anomaly detection (TSAD) has attracted considerable scholarly and
industrial interest. However, existing algorithms exhibit a gap in terms of
training paradigm, online detection paradigm, and evaluation criteria when
compared to the actual needs of real-world industrial systems. Firstly, current
algorithms typically train a specific model for each individual time series. In
a large-scale online system with tens of thousands of curves, maintaining such
a multitude of models is impractical. The performance of using merely one
single unified model to detect anomalies remains unknown. Secondly, most TSAD
models are trained on the historical part of a time series and are tested on
its future segment. In distributed systems, however, there are frequent system
deployments and upgrades, with new, previously unseen time series emerging
daily. The performance of testing newly incoming unseen time series on current
TSAD algorithms remains unknown. Lastly, although some papers have conducted
detailed surveys, the absence of an online evaluation platform prevents
answering questions like "Who is the best at anomaly detection at the current
stage?" In this paper, we propose TimeSeriesBench, an industrial-grade
benchmark that we continuously maintain as a leaderboard. On this leaderboard,
we assess the performance of existing algorithms across more than 168
evaluation settings combining different training and testing paradigms,
evaluation metrics and datasets. Through our comprehensive analysis of the
results, we provide recommendations for the future design of anomaly detection
algorithms. To address known issues with existing public datasets, we release
an industrial dataset to the public together with TimeSeriesBench. All code,
data, and the online leaderboard have been made publicly available.
Related papers
- Deep Time Series Models: A Comprehensive Survey and Benchmark [74.28364194333447]
Time series data is of great significance in real-world scenarios.
Recent years have witnessed remarkable breakthroughs in the time series community.
We release Time Series Library (TSLib) as a fair benchmark of deep time series models for diverse analysis tasks.
arXiv Detail & Related papers (2024-07-18T08:31:55Z) - TSI-Bench: Benchmarking Time Series Imputation [52.27004336123575]
TSI-Bench is a comprehensive benchmark suite for time series imputation utilizing deep learning techniques.
The TSI-Bench pipeline standardizes experimental settings to enable fair evaluation of imputation algorithms.
TSI-Bench innovatively provides a systematic paradigm to tailor time series forecasting algorithms for imputation purposes.
arXiv Detail & Related papers (2024-06-18T16:07:33Z) - Graph Spatiotemporal Process for Multivariate Time Series Anomaly
Detection with Missing Values [67.76168547245237]
We introduce a novel framework called GST-Pro, which utilizes a graphtemporal process and anomaly scorer to detect anomalies.
Our experimental results show that the GST-Pro method can effectively detect anomalies in time series data and outperforms state-of-the-art methods.
arXiv Detail & Related papers (2024-01-11T10:10:16Z) - Making the End-User a Priority in Benchmarking: OrionBench for
Unsupervised Time Series Anomaly Detection [9.054540533394924]
Time series anomaly detection is a prevalent problem in many application domains such as patient monitoring in healthcare, forecasting in finance, or predictive maintenance in energy.
We propose OrionBench -- a user centric continuously maintained benchmark for unsupervised time series anomaly detection.
We demonstrate the usage of OrionBench, and the progression of pipelines across 16 releases published over the course of three years.
arXiv Detail & Related papers (2023-10-26T19:43:16Z) - Online Evolutionary Neural Architecture Search for Multivariate
Non-Stationary Time Series Forecasting [72.89994745876086]
This work presents the Online Neuro-Evolution-based Neural Architecture Search (ONE-NAS) algorithm.
ONE-NAS is a novel neural architecture search method capable of automatically designing and dynamically training recurrent neural networks (RNNs) for online forecasting tasks.
Results demonstrate that ONE-NAS outperforms traditional statistical time series forecasting methods.
arXiv Detail & Related papers (2023-02-20T22:25:47Z) - Deep Generative model with Hierarchical Latent Factors for Time Series
Anomaly Detection [40.21502451136054]
This work presents DGHL, a new family of generative models for time series anomaly detection.
A top-down Convolution Network maps a novel hierarchical latent space to time series windows, exploiting temporal dynamics to encode information efficiently.
Our method outperformed current state-of-the-art models on four popular benchmark datasets.
arXiv Detail & Related papers (2022-02-15T17:19:44Z) - Networked Time Series Prediction with Incomplete Data [59.45358694862176]
We propose NETS-ImpGAN, a novel deep learning framework that can be trained on incomplete data with missing values in both history and future.
We conduct extensive experiments on three real-world datasets under different missing patterns and missing rates.
arXiv Detail & Related papers (2021-10-05T18:20:42Z) - TadGAN: Time Series Anomaly Detection Using Generative Adversarial
Networks [73.01104041298031]
TadGAN is an unsupervised anomaly detection approach built on Generative Adversarial Networks (GANs)
To capture the temporal correlations of time series, we use LSTM Recurrent Neural Networks as base models for Generators and Critics.
To demonstrate the performance and generalizability of our approach, we test several anomaly scoring techniques and report the best-suited one.
arXiv Detail & Related papers (2020-09-16T15:52:04Z) - Automated Model Selection for Time-Series Anomaly Detection [6.396011708581161]
Many companies need to monitor thousands of temporal signals for their applications and services and require instant feedback and alerts for potential incidents in time.
The task is challenging because of the complex characteristics of time-series, which are messy, and often without proper labels.
This prohibits training supervised models because of lack of labels and a single model hardly fits different time series.
We present an automated model selection framework to automatically find the most suitable detection model with proper parameters for the incoming data.
arXiv Detail & Related papers (2020-08-25T07:23:43Z) - Anomaly Detection at Scale: The Case for Deep Distributional Time Series
Models [14.621700495712647]
Main novelty in our approach is that instead of modeling time series consisting of real values or vectors of real values, we model time series of probability distributions over real values (or vectors)
Our method is amenable to streaming anomaly detection and scales to monitoring for anomalies on millions of time series.
We show that we outperform popular open-source anomaly detection tools by up to 17% average improvement for a real-world data set.
arXiv Detail & Related papers (2020-07-30T15:48:55Z) - RobustTAD: Robust Time Series Anomaly Detection via Decomposition and
Convolutional Neural Networks [37.16594704493679]
We propose RobustTAD, a Robust Time series Anomaly Detection framework.
It integrates robust seasonal-trend decomposition and convolutional neural network for time series data.
It is deployed as a public online service and widely adopted in different business scenarios at Alibaba Group.
arXiv Detail & Related papers (2020-02-21T20:43:45Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.