Evaluation Strategy of Time-series Anomaly Detection with Decay Function

• URL: http://arxiv.org/abs/2305.09691v1
• Date: Mon, 15 May 2023 23:55:49 GMT
• Title: Evaluation Strategy of Time-series Anomaly Detection with Decay Function
• Authors: Yongwan Gim, Kyushik Min
• Abstract summary: We propose a novel evaluation protocol called the Point-Adjusted protocol with decay function (PAdf) to evaluate the time-series anomaly detection algorithm. This paper theoretically and experimentally shows that the PAdf protocol solves the over- and under-estimation problems of existing protocols.
• Score: 1.713291434132985
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent algorithms of time-series anomaly detection have been evaluated by applying a Point Adjustment (PA) protocol. However, the PA protocol has a problem of overestimating the performance of the detection algorithms because it only depends on the number of detected abnormal segments and their size. We propose a novel evaluation protocol called the Point-Adjusted protocol with decay function (PAdf) to evaluate the time-series anomaly detection algorithm by reflecting the following ideal requirements: detect anomalies quickly and accurately without false alarms. This paper theoretically and experimentally shows that the PAdf protocol solves the over- and under-estimation problems of existing protocols such as PA and PA\%K. By conducting re-evaluations of SOTA models in benchmark datasets, we show that the PA protocol only focuses on finding many anomalous segments, whereas the score of the PAdf protocol considers not only finding many segments but also detecting anomalies quickly without delay.

Related papers

• PaAno: Patch-Based Representation Learning for Time-Series Anomaly Detection [14.610038284393164]
  We propose Patch-based representation learning for time-series anomaly detection (PaAno)<n>PaAno extracts short temporal patches from time-series training data and uses a 1D convolutional neural network to embed each patch into a vector representation.<n> evaluated on the TSB-AD benchmark, PaAno achieved state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-02-01T17:52:59Z)
• Towards Unbiased Evaluation of Time-series Anomaly Detector [6.521243384420707]
  Time series anomaly detection (TSAD) is an evolving area of research motivated by its critical applications. In this work, we propose an alternative adjustment protocol called Balanced point adjustment'' (BA)
  arXiv  Detail & Related papers  (2024-09-19T19:02:45Z)
• PATE: Proximity-Aware Time series anomaly Evaluation [3.0377067713090633]
  Traditional performance metrics assume iid data and fail to capture the complex temporal dynamics and specific characteristics of time series anomalies. We introduce Proximity-Aware Time series anomaly Evaluation (PATE), a novel evaluation metric that incorporates the temporal relationship between prediction and anomaly intervals. Our experiments with synthetic and real-world datasets show the superiority of PATE in providing more sensible and accurate evaluations.
  arXiv  Detail & Related papers  (2024-05-20T15:06:36Z)
• Graph Spatiotemporal Process for Multivariate Time Series Anomaly Detection with Missing Values [67.76168547245237]
  We introduce a novel framework called GST-Pro, which utilizes a graphtemporal process and anomaly scorer to detect anomalies. Our experimental results show that the GST-Pro method can effectively detect anomalies in time series data and outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-01-11T10:10:16Z)
• Video Anomaly Detection via Spatio-Temporal Pseudo-Anomaly Generation : A Unified Approach [49.995833831087175]
  This work proposes a novel method for generating generic Video-temporal PAs by inpainting a masked out region of an image. In addition, we present a simple unified framework to detect real-world anomalies under the OCC setting. Our method performs on par with other existing state-of-the-art PAs generation and reconstruction based methods under the OCC setting.
  arXiv  Detail & Related papers  (2023-11-27T13:14:06Z)
• Beyond AUROC & co. for evaluating out-of-distribution detection performance [50.88341818412508]
  Given their relevance for safe(r) AI, it is important to examine whether the basis for comparing OOD detection methods is consistent with practical needs. We propose a new metric - Area Under the Threshold Curve (AUTC), which explicitly penalizes poor separation between ID and OOD samples.
  arXiv  Detail & Related papers  (2023-06-26T12:51:32Z)
• Are we certain it's anomalous? [57.729669157989235]
  Anomaly detection in time series is a complex task since anomalies are rare due to highly non-linear temporal correlations. Here we propose the novel use of Hyperbolic uncertainty for Anomaly Detection (HypAD) HypAD learns self-supervisedly to reconstruct the input signal.
  arXiv  Detail & Related papers  (2022-11-16T21:31:39Z)
• Towards a Rigorous Evaluation of Time-series Anomaly Detection [15.577148857778484]
  In recent years, proposed studies on time-series anomaly detection (TAD) report high F1 scores on benchmark TAD datasets. Most studies apply a peculiar evaluation protocol called point adjustment (PA) before scoring. In this paper, we reveal that the PA protocol has a great possibility of overestimating the detection performance.
  arXiv  Detail & Related papers  (2021-09-11T11:14:04Z)
• Anomaly Detection in Predictive Maintenance: A New Evaluation Framework for Temporal Unsupervised Anomaly Detection Algorithms [9.316869851584771]
  The research in anomaly detection lacks a unified definition of what represents an anomalous instance. We propose a concept of positive and negative instances to intervals to be able to evaluate unsupervised anomaly detection algorithms. To show the usefulness of this definition, we include a case study of Big Data algorithms with a real-world time-series problem provided by the company ArcelorMittal.
  arXiv  Detail & Related papers  (2021-05-26T20:15:40Z)
• Round-robin differential phase-time-shifting protocol for quantum key distribution: theory and experiment [58.03659958248968]
  Quantum key distribution (QKD) allows the establishment of common cryptographic keys among distant parties. Recently, a QKD protocol that circumvents the need for monitoring signal disturbance, has been proposed and demonstrated in initial experiments. We derive the security proofs of the round-robin differential phase-time-shifting protocol in the collective attack scenario. Our results show that the RRDPTS protocol can achieve higher secret key rate in comparison with the RRDPS, in the condition of high quantum bit error rate.
  arXiv  Detail & Related papers  (2021-03-15T15:20:09Z)
• How Far Should We Look Back to Achieve Effective Real-Time Time-Series Anomaly Detection? [1.0437764544103274]
  Anomaly detection is the process of identifying unexpected events or ab-normalities in data. RePAD (Real-time Proactive Anomaly Detection algorithm) is a generic approach with all above-mentioned features. It is unclear how different amounts of historical data points affect the performance of RePAD.
  arXiv  Detail & Related papers  (2021-02-12T14:51:05Z)
• Change Point Detection in Time Series Data using Autoencoders with a Time-Invariant Representation [69.34035527763916]
  Change point detection (CPD) aims to locate abrupt property changes in time series data. Recent CPD methods demonstrated the potential of using deep learning techniques, but often lack the ability to identify more subtle changes in the autocorrelation statistics of the signal. We employ an autoencoder-based methodology with a novel loss function, through which the used autoencoders learn a partially time-invariant representation that is tailored for CPD.
  arXiv  Detail & Related papers  (2020-08-21T15:03:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.