VUS: Effective and Efficient Accuracy Measures for Time-Series Anomaly Detection

• URL: http://arxiv.org/abs/2502.13318v1
• Date: Tue, 18 Feb 2025 22:19:52 GMT
• Title: VUS: Effective and Efficient Accuracy Measures for Time-Series Anomaly Detection
• Authors: Paul Boniol, Ashwin K. Krishna, Marine Bruel, Qinghua Liu, Mingyi Huang, Themis Palpanas, Ruey S. Tsay, Aaron Elmore, Michael J. Franklin, John Paparrizos,
• Abstract summary: This paper extensively evaluates quality measures for time-series AD to assess their robustness under noise, misalignments, and different anomaly cardinality ratios.<n>Our results indicate that measures producing quality values independently of a threshold are more suitable for time-series AD.
• Score: 17.751395424719167
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Anomaly detection (AD) is a fundamental task for time-series analytics with important implications for the downstream performance of many applications. In contrast to other domains where AD mainly focuses on point-based anomalies (i.e., outliers in standalone observations), AD for time series is also concerned with range-based anomalies (i.e., outliers spanning multiple observations). Nevertheless, it is common to use traditional point-based information retrieval measures, such as Precision, Recall, and F-score, to assess the quality of methods by thresholding the anomaly score to mark each point as an anomaly or not. However, mapping discrete labels into continuous data introduces unavoidable shortcomings, complicating the evaluation of range-based anomalies. Notably, the choice of evaluation measure may significantly bias the experimental outcome. Despite over six decades of attention, there has never been a large-scale systematic quantitative and qualitative analysis of time-series AD evaluation measures. This paper extensively evaluates quality measures for time-series AD to assess their robustness under noise, misalignments, and different anomaly cardinality ratios. Our results indicate that measures producing quality values independently of a threshold (i.e., AUC-ROC and AUC-PR) are more suitable for time-series AD. Motivated by this observation, we first extend the AUC-based measures to account for range-based anomalies. Then, we introduce a new family of parameter-free and threshold-independent measures, Volume Under the Surface (VUS), to evaluate methods while varying parameters. We also introduce two optimized implementations for VUS that reduce significantly the execution time of the initial implementation. Our findings demonstrate that our four measures are significantly more robust in assessing the quality of time-series AD methods.

Related papers

• CCE: Confidence-Consistency Evaluation for Time Series Anomaly Detection [56.302586730134806]
  We introduce Confidence-Consistency Evaluation (CCE), a novel evaluation metric.<n>CCE simultaneously measures prediction confidence and uncertainty consistency.<n>We also establish RankEval, a benchmark for comparing the ranking capabilities of various metrics.
  arXiv  Detail & Related papers  (2025-09-01T03:38:38Z)
• Rethinking Metrics and Benchmarks of Video Anomaly Detection [58.37571339811799]
  Video Anomaly Detection (VAD) aims to detect anomalies that deviate from expectation.<n>Existing VAD metrics are influenced by single annotation bias.<n>Existing benchmarks lack the capability to evaluate scene overfitting of fully/weakly-supervised algorithms.
  arXiv  Detail & Related papers  (2025-05-25T08:09:42Z)
• OIPR: Evaluation for Time-series Anomaly Detection Inspired by Operator Interest [26.460594836601004]
  We propose a novel set of time-series anomaly detection evaluation metrics, called OIPR. OIPR models the process of operators receiving detector alarms and handling faults, utilizing area under the operator interest curve to evaluate the performance of TAD algorithms. It achieves a balance between point and event perspectives, overcoming their primary limitations and offering applicability to broader situations.
  arXiv  Detail & Related papers  (2025-03-03T07:37:24Z)
• Towards Unbiased Evaluation of Time-series Anomaly Detector [6.521243384420707]
  Time series anomaly detection (TSAD) is an evolving area of research motivated by its critical applications. In this work, we propose an alternative adjustment protocol called Balanced point adjustment'' (BA)
  arXiv  Detail & Related papers  (2024-09-19T19:02:45Z)
• PATE: Proximity-Aware Time series anomaly Evaluation [3.0377067713090633]
  Traditional performance metrics assume iid data and fail to capture the complex temporal dynamics and specific characteristics of time series anomalies. We introduce Proximity-Aware Time series anomaly Evaluation (PATE), a novel evaluation metric that incorporates the temporal relationship between prediction and anomaly intervals. Our experiments with synthetic and real-world datasets show the superiority of PATE in providing more sensible and accurate evaluations.
  arXiv  Detail & Related papers  (2024-05-20T15:06:36Z)
• Learning Feature Inversion for Multi-class Anomaly Detection under General-purpose COCO-AD Benchmark [101.23684938489413]
  Anomaly detection (AD) is often focused on detecting anomalies for industrial quality inspection and medical lesion examination. This work first constructs a large-scale and general-purpose COCO-AD dataset by extending COCO to the AD field. Inspired by the metrics in the segmentation field, we propose several more practical threshold-dependent AD-specific metrics.
  arXiv  Detail & Related papers  (2024-04-16T17:38:26Z)
• Unraveling the "Anomaly" in Time Series Anomaly Detection: A Self-supervised Tri-domain Solution [89.16750999704969]
  Anomaly labels hinder traditional supervised models in time series anomaly detection. Various SOTA deep learning techniques, such as self-supervised learning, have been introduced to tackle this issue. We propose a novel self-supervised learning based Tri-domain Anomaly Detector (TriAD)
  arXiv  Detail & Related papers  (2023-11-19T05:37:18Z)
• CARLA: Self-supervised Contrastive Representation Learning for Time Series Anomaly Detection [53.83593870825628]
  One main challenge in time series anomaly detection (TSAD) is the lack of labelled data in many real-life scenarios. Most of the existing anomaly detection methods focus on learning the normal behaviour of unlabelled time series in an unsupervised manner. We introduce a novel end-to-end self-supervised ContrAstive Representation Learning approach for time series anomaly detection.
  arXiv  Detail & Related papers  (2023-08-18T04:45:56Z)
• Unsupervised Anomaly Detection in Time-series: An Extensive Evaluation and Analysis of State-of-the-art Methods [10.618572317896515]
  Unsupervised anomaly detection in time-series has been extensively investigated in the literature. This paper proposes an in-depth evaluation study of recent unsupervised anomaly detection techniques in time-series.
  arXiv  Detail & Related papers  (2022-12-06T15:05:54Z)
• TiSAT: Time Series Anomaly Transformer [30.68108039722565]
  We show that a rudimentary Random Guess method can outperform state-of-the-art detectors in terms of this popular but faulty evaluation criterion. In this work, we propose a proper evaluation metric that measures the timeliness and precision of detecting sequential anomalies.
  arXiv  Detail & Related papers  (2022-03-10T05:46:58Z)
• Data-Efficient and Interpretable Tabular Anomaly Detection [54.15249463477813]
  We propose a novel framework that adapts a white-box model class, Generalized Additive Models, to detect anomalies. In addition, the proposed framework, DIAD, can incorporate a small amount of labeled data to further boost anomaly detection performances in semi-supervised settings.
  arXiv  Detail & Related papers  (2022-03-03T22:02:56Z)
• TadGAN: Time Series Anomaly Detection Using Generative Adversarial Networks [73.01104041298031]
  TadGAN is an unsupervised anomaly detection approach built on Generative Adversarial Networks (GANs) To capture the temporal correlations of time series, we use LSTM Recurrent Neural Networks as base models for Generators and Critics. To demonstrate the performance and generalizability of our approach, we test several anomaly scoring techniques and report the best-suited one.
  arXiv  Detail & Related papers  (2020-09-16T15:52:04Z)
• Uncertainty-aware Score Distribution Learning for Action Quality Assessment [91.05846506274881]
  We propose an uncertainty-aware score distribution learning (USDL) approach for action quality assessment (AQA) Specifically, we regard an action as an instance associated with a score distribution, which describes the probability of different evaluated scores. Under the circumstance where fine-grained score labels are available, we devise a multi-path uncertainty-aware score distributions learning (MUSDL) method to explore the disentangled components of a score.
  arXiv  Detail & Related papers  (2020-06-13T15:41:29Z)
• Deep Recurrent Model for Individualized Prediction of Alzheimer's Disease Progression [4.034948808542701]
  Alzheimer's disease (AD) is one of the major causes of dementia and is characterized by slow progression over several years. We propose a novel computational framework that can predict the phenotypic measurements of MRI biomarkers and trajectories of clinical status.
  arXiv  Detail & Related papers  (2020-05-06T08:08:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.