A Problem-Oriented Taxonomy of Evaluation Metrics for Time Series Anomaly Detection

• URL: http://arxiv.org/abs/2511.18739v1
• Date: Mon, 24 Nov 2025 04:09:04 GMT
• Title: A Problem-Oriented Taxonomy of Evaluation Metrics for Time Series Anomaly Detection
• Abstract summary: Time series anomaly detection is widely used in IoT and cyber-physical systems.<n>This study introduces a problem-oriented framework that reinterprets existing metrics based on the specific evaluation challenges they are designed to address.
• Score: 11.114573387946498
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Time series anomaly detection is widely used in IoT and cyber-physical systems, yet its evaluation remains challenging due to diverse application objectives and heterogeneous metric assumptions. This study introduces a problem-oriented framework that reinterprets existing metrics based on the specific evaluation challenges they are designed to address, rather than their mathematical forms or output structures. We categorize over twenty commonly used metrics into six dimensions: 1) basic accuracy-driven evaluation; 2) timeliness-aware reward mechanisms; 3) tolerance to labeling imprecision; 4) penalties reflecting human-audit cost; 5) robustness against random or inflated scores; and 6) parameter-free comparability for cross-dataset benchmarking. Comprehensive experiments are conducted to examine metric behavior under genuine, random, and oracle detection scenarios. By comparing their resulting score distributions, we quantify each metric's discriminative ability -- its capability to distinguish meaningful detections from random noise. The results show that while most event-level metrics exhibit strong separability, several widely used metrics (e.g., NAB, Point-Adjust) demonstrate limited resistance to random-score inflation. These findings reveal that metric suitability must be inherently task-dependent and aligned with the operational objectives of IoT applications. The proposed framework offers a unified analytical perspective for understanding existing metrics and provides practical guidance for selecting or developing more context-aware, robust, and fair evaluation methodologies for time series anomaly detection.

Related papers

• DQE: A Semantic-Aware Evaluation Metric for Time Series Anomaly Detection [10.700735533120257]
  We revisit the evaluation of time series anomaly detection from the perspective of detection semantics.<n>We introduce a partitioning strategy grounded in detection semantics, which decomposes the local temporal region of each anomaly into three functionally distinct subregions.<n>Using this partitioning, we evaluate overall detection behavior across events and design finer-grained scoring mechanisms for each subregion, enabling more reliable and interpretable assessment.
  arXiv  Detail & Related papers  (2026-03-06T10:38:51Z)
• Formally Exploring Time-Series Anomaly Detection Evaluation Metrics [30.10218977177152]
  Undetected anomalies in time series can trigger catastrophic failures in safety-critical systems.<n>Current metrics capture only narrow aspects of the task and often yield misleading results.<n>We introduce verifiable properties that formalize essential requirements for evaluating time-series anomaly detection.
  arXiv  Detail & Related papers  (2025-10-20T14:10:39Z)
• The illusion of a perfect metric: Why evaluating AI's words is harder than it looks [0.0]
  Natural Language Generation (NLG) is crucial for the practical adoption of AI.<n>Human evaluation is considered the de-facto standard, but it is expensive and lacks scalability.<n>No single metric has emerged as a definitive solution, resulting in studies using different ones without fully considering the implications.
  arXiv  Detail & Related papers  (2025-08-19T13:22:41Z)
• Position: All Current Generative Fidelity and Diversity Metrics are Flawed [58.815519650465774]
  We show that all current generative fidelity and diversity metrics are flawed.<n>Our aim is to convince the research community to spend more effort in developing metrics, instead of models.
  arXiv  Detail & Related papers  (2025-05-28T15:10:33Z)
• OIPR: Evaluation for Time-series Anomaly Detection Inspired by Operator Interest [26.460594836601004]
  We propose a novel set of time-series anomaly detection evaluation metrics, called OIPR.<n>OIPR models the process of operators receiving detector alarms and handling faults, utilizing area under the operator interest curve to evaluate the performance of TAD algorithms.<n>It achieves a balance between point and event perspectives, overcoming their primary limitations and offering applicability to broader situations.
  arXiv  Detail & Related papers  (2025-03-03T07:37:24Z)
• Open-set object detection: towards unified problem formulation and benchmarking [2.4374097382908477]
  We introduce two benchmarks: a unified VOC-COCO evaluation, and the new OpenImagesRoad benchmark which provides clear hierarchical object definition besides new evaluation metrics. State-of-the-art methods are extensively evaluated on the proposed benchmarks. This study provides a clear problem definition, ensures consistent evaluations, and draws new conclusions about effectiveness of OSOD strategies.
  arXiv  Detail & Related papers  (2024-11-08T13:40:01Z)
• A Comprehensive Library for Benchmarking Multi-class Visual Anomaly Detection [89.92916473403108]
  This paper proposes a comprehensive visual anomaly detection benchmark, ADer, which is a modular framework for new methods.<n>The benchmark includes multiple datasets from industrial and medical domains, implementing fifteen state-of-the-art methods and nine comprehensive metrics.<n>We objectively reveal the strengths and weaknesses of different methods and provide insights into the challenges and future directions of multi-class visual anomaly detection.
  arXiv  Detail & Related papers  (2024-06-05T13:40:07Z)
• On Pixel-level Performance Assessment in Anomaly Detection [87.7131059062292]
  Anomaly detection methods have demonstrated remarkable success across various applications. However, assessing their performance, particularly at the pixel-level, presents a complex challenge. In this paper, we dissect the intricacies of this challenge, underscored by visual evidence and statistical analysis.
  arXiv  Detail & Related papers  (2023-10-25T08:02:27Z)
• Identifying Performance Issues in Cloud Service Systems Based on Relational-Temporal Features [11.83269525626691]
  Cloud systems are susceptible to performance issues, which may cause service-level agreement violations and financial losses. We propose a learning-based approach that leverages both the relational and temporal features of metrics to identify performance issues.
  arXiv  Detail & Related papers  (2023-07-20T13:41:26Z)
• Navigating the Metric Maze: A Taxonomy of Evaluation Metrics for Anomaly Detection in Time Series [0.456877715768796]
  This paper provides a comprehensive overview of the metrics used for the evaluation of time series anomaly detection methods. Twenty metrics are analyzed and discussed in detail, highlighting the unique suitability of each for specific tasks.
  arXiv  Detail & Related papers  (2023-03-02T13:58:06Z)
• ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning [63.77667876176978]
  Large language models show improved downstream task interpretability when prompted to generate step-by-step reasoning to justify their final answers. These reasoning steps greatly improve model interpretability and verification, but objectively studying their correctness is difficult. We present ROS, a suite of interpretable, unsupervised automatic scores that improve and extend previous text generation evaluation metrics.
  arXiv  Detail & Related papers  (2022-12-15T15:52:39Z)
• GO FIGURE: A Meta Evaluation of Factuality in Summarization [131.1087461486504]
  We introduce GO FIGURE, a meta-evaluation framework for evaluating factuality evaluation metrics. Our benchmark analysis on ten factuality metrics reveals that our framework provides a robust and efficient evaluation. It also reveals that while QA metrics generally improve over standard metrics that measure factuality across domains, performance is highly dependent on the way in which questions are generated.
  arXiv  Detail & Related papers  (2020-10-24T08:30:20Z)
• Robust Learning Through Cross-Task Consistency [92.42534246652062]
  We propose a broadly applicable and fully computational method for augmenting learning with Cross-Task Consistency. We observe that learning with cross-task consistency leads to more accurate predictions and better generalization to out-of-distribution inputs.
  arXiv  Detail & Related papers  (2020-06-07T09:24:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.