We Need to Rethink Benchmarking in Anomaly Detection

• URL: http://arxiv.org/abs/2507.15584v1
• Date: Mon, 21 Jul 2025 13:02:49 GMT
• Title: We Need to Rethink Benchmarking in Anomaly Detection
• Authors: Philipp Röchner, Simon Klüttermann, Franz Rothlauf, Daniel Schlör,
• Abstract summary: We argue that this stagnation is due to limitations in how we evaluate anomaly detection algorithms.<n>In our opinion, anomaly detection should be studied using scenarios that capture the relevant characteristics of different applications.
• Score: 0.9124662097191378
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Despite the continuous proposal of new anomaly detection algorithms and extensive benchmarking efforts, progress seems to stagnate, with only minor performance differences between established baselines and new algorithms. In this position paper, we argue that this stagnation is due to limitations in how we evaluate anomaly detection algorithms. Current benchmarking does not, for example, sufficiently reflect the diversity of anomalies in applications ranging from predictive maintenance to scientific discovery. Consequently, we need to rethink benchmarking in anomaly detection. In our opinion, anomaly detection should be studied using scenarios that capture the relevant characteristics of different applications. We identify three key areas for improvement: First, we need to identify anomaly detection scenarios based on a common taxonomy. Second, anomaly detection pipelines should be analyzed end-to-end and by component. Third, evaluating anomaly detection algorithms should be meaningful regarding the scenario's objectives.

Related papers

• Self-supervised Feature Adaptation for 3D Industrial Anomaly Detection [59.41026558455904]
  We focus on multi-modal anomaly detection. Specifically, we investigate early multi-modal approaches that attempted to utilize models pre-trained on large-scale visual datasets. We propose a Local-to-global Self-supervised Feature Adaptation (LSFA) method to finetune the adaptors and learn task-oriented representation toward anomaly detection.
  arXiv  Detail & Related papers  (2024-01-06T07:30:41Z)
• Don't Miss Out on Novelty: Importance of Novel Features for Deep Anomaly Detection [64.21963650519312]
  Anomaly Detection (AD) is a critical task that involves identifying observations that do not conform to a learned model of normality. We propose a novel approach to AD using explainability to capture such novel features as unexplained observations in the input space. Our approach establishes a new state-of-the-art across multiple benchmarks, handling diverse anomaly types.
  arXiv  Detail & Related papers  (2023-10-01T21:24:05Z)
• Precursor-of-Anomaly Detection for Irregular Time Series [31.73234935455713]
  We present a novel type of anomaly detection, called Precursor-of-Anomaly (PoA) detection. To solve both problems at the same time, we present a neural controlled differential equation-based neural network and its multi-task learning algorithm.
  arXiv  Detail & Related papers  (2023-06-27T14:10:09Z)
• PULL: Reactive Log Anomaly Detection Based On Iterative PU Learning [58.85063149619348]
  We propose PULL, an iterative log analysis method for reactive anomaly detection based on estimated failure time windows. Our evaluation shows that PULL consistently outperforms ten benchmark baselines across three different datasets.
  arXiv  Detail & Related papers  (2023-01-25T16:34:43Z)
• Towards Interpretable Anomaly Detection via Invariant Rule Mining [2.538209532048867]
  In this work, we pursue highly interpretable anomaly detection via invariant rule mining. Specifically, we leverage decision tree learning and association rule mining to automatically generate invariant rules. The generated invariant rules can provide explicit explanation of anomaly detection results and thus are extremely useful for subsequent decision-making.
  arXiv  Detail & Related papers  (2022-11-24T13:03:20Z)
• Deep Learning for Time Series Anomaly Detection: A Survey [53.83593870825628]
  Time series anomaly detection has applications in a wide range of research fields and applications, including manufacturing and healthcare. The large size and complex patterns of time series have led researchers to develop specialised deep learning models for detecting anomalous patterns. This survey focuses on providing structured and comprehensive state-of-the-art time series anomaly detection models through the use of deep learning.
  arXiv  Detail & Related papers  (2022-11-09T22:40:22Z)
• Explainable Deep Few-shot Anomaly Detection with Deviation Networks [123.46611927225963]
  We introduce a novel weakly-supervised anomaly detection framework to train detection models. The proposed approach learns discriminative normality by leveraging the labeled anomalies and a prior probability. Our model is substantially more sample-efficient and robust, and performs significantly better than state-of-the-art competing methods in both closed-set and open-set settings.
  arXiv  Detail & Related papers  (2021-08-01T14:33:17Z)
• Anomaly Detection in Predictive Maintenance: A New Evaluation Framework for Temporal Unsupervised Anomaly Detection Algorithms [9.316869851584771]
  The research in anomaly detection lacks a unified definition of what represents an anomalous instance. We propose a concept of positive and negative instances to intervals to be able to evaluate unsupervised anomaly detection algorithms. To show the usefulness of this definition, we include a case study of Big Data algorithms with a real-world time-series problem provided by the company ArcelorMittal.
  arXiv  Detail & Related papers  (2021-05-26T20:15:40Z)
• ESAD: End-to-end Deep Semi-supervised Anomaly Detection [85.81138474858197]
  We propose a new objective function that measures the KL-divergence between normal and anomalous data. The proposed method significantly outperforms several state-of-the-arts on multiple benchmark datasets.
  arXiv  Detail & Related papers  (2020-12-09T08:16:35Z)
• Algorithmic Frameworks for the Detection of High Density Anomalies [0.0]
  High-density anomalies are deviant cases positioned in the most normal regions of the data space. This study introduces several non-parametric algorithmic frameworks for unsupervised detection.
  arXiv  Detail & Related papers  (2020-10-09T17:48:02Z)
• Anomaly Detection in Univariate Time-series: A Survey on the State-of-the-Art [0.0]
  Anomaly detection for time-series data has been an important research field for a long time. Recent years an increasing number of machine learning algorithms have been developed to detect anomalies on time-series. Researchers tried to improve these techniques using (deep) neural networks.
  arXiv  Detail & Related papers  (2020-04-01T13:22:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.