Exploring the Impact of Outlier Variability on Anomaly Detection Evaluation Metrics

• URL: http://arxiv.org/abs/2409.15986v1
• Date: Tue, 24 Sep 2024 11:39:09 GMT
• Title: Exploring the Impact of Outlier Variability on Anomaly Detection Evaluation Metrics
• Authors: Minjae Ok, Simon Klüttermann, Emmanuel Müller,
• Abstract summary: This study focuses on examining the behaviors of three widely used anomaly detection metrics under different conditions. We present findings that challenge the conventional understanding of these metrics and reveal nuanced behaviors under varying conditions. The results of our study contribute to a more refined understanding of metric selection and interpretation in anomaly detection, offering valuable insights for both researchers and practitioners in the field.
• Score: 4.943054375935879
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Anomaly detection is a dynamic field, in which the evaluation of models plays a critical role in understanding their effectiveness. The selection and interpretation of the evaluation metrics are pivotal, particularly in scenarios with varying amounts of anomalies. This study focuses on examining the behaviors of three widely used anomaly detection metrics under different conditions: F1 score, Receiver Operating Characteristic Area Under Curve (ROC AUC), and Precision-Recall Curve Area Under Curve (AUCPR). Our study critically analyzes the extent to which these metrics provide reliable and distinct insights into model performance, especially considering varying levels of outlier fractions and contamination thresholds in datasets. Through a comprehensive experimental setup involving widely recognized algorithms for anomaly detection, we present findings that challenge the conventional understanding of these metrics and reveal nuanced behaviors under varying conditions. We demonstrated that while the F1 score and AUCPR are sensitive to outlier fractions, the ROC AUC maintains consistency and is unaffected by such variability. Additionally, under conditions of a fixed outlier fraction in the test set, we observe an alignment between ROC AUC and AUCPR, indicating that the choice between these two metrics may be less critical in such scenarios. The results of our study contribute to a more refined understanding of metric selection and interpretation in anomaly detection, offering valuable insights for both researchers and practitioners in the field.

Related papers

• Dealing with Uncertainty in Contextual Anomaly Detection [14.492457340456737]
  Contextual anomaly detection (CAD) aims to identify anomalies in a target (behavioral) variable conditioned on a set of contextual variables.<n>We propose a novel framework for CAD, normalcy score (NS), that explicitly models both the aleatoric and epistemic uncertainties.<n>We demonstrate that NS outperforms state-of-the-art CAD methods in both detection accuracy and interpretability.
  arXiv  Detail & Related papers  (2025-07-06T18:02:11Z)
• Performance Metric for Multiple Anomaly Score Distributions with Discrete Severity Levels [4.66313002591741]
  We propose a weighted sum of the area under the receiver operating characteristic curve (WS-AUROC) for classifying severity levels based on anomaly scores. We also propose an anomaly detector that achieves clear separation of distributions and outperforms the ablation models on the WS-AUROC and AUROC metrics.
  arXiv  Detail & Related papers  (2024-08-09T02:17:49Z)
• Harnessing Feature Clustering For Enhanced Anomaly Detection With Variational Autoencoder And Dynamic Threshold [0.0]
  We introduce an anomaly detection method to identify critical periods and features influencing extreme climate events like snowmelt in the Arctic. This method leverages the Variational Autoencoder integrated with dynamic thresholding and correlation-based feature clustering. This framework enhances the VAE's ability to identify localized dependencies and learn the temporal relationships in climate data.
  arXiv  Detail & Related papers  (2024-07-14T01:52:10Z)
• Challenges and Considerations in the Evaluation of Bayesian Causal Discovery [49.0053848090947]
  Representing uncertainty in causal discovery is a crucial component for experimental design, and more broadly, for safe and reliable causal decision making. Unlike non-Bayesian causal discovery, which relies on a single estimated causal graph and model parameters for assessment, causal discovery presents challenges due to the nature of its quantity. No consensus on the most suitable metric for evaluation.
  arXiv  Detail & Related papers  (2024-06-05T12:45:23Z)
• Meta-Learners for Partially-Identified Treatment Effects Across Multiple Environments [67.80453452949303]
  Estimating the conditional average treatment effect (CATE) from observational data is relevant for many applications such as personalized medicine. Here, we focus on the widespread setting where the observational data come from multiple environments. We propose different model-agnostic learners (so-called meta-learners) to estimate the bounds that can be used in combination with arbitrary machine learning models.
  arXiv  Detail & Related papers  (2024-06-04T16:31:43Z)
• Unraveling the "Anomaly" in Time Series Anomaly Detection: A Self-supervised Tri-domain Solution [89.16750999704969]
  Anomaly labels hinder traditional supervised models in time series anomaly detection. Various SOTA deep learning techniques, such as self-supervised learning, have been introduced to tackle this issue. We propose a novel self-supervised learning based Tri-domain Anomaly Detector (TriAD)
  arXiv  Detail & Related papers  (2023-11-19T05:37:18Z)
• An Iterative Method for Unsupervised Robust Anomaly Detection Under Data Contamination [24.74938110451834]
  Most deep anomaly detection models are based on learning normality from datasets. In practice, the normality assumption is often violated due to the nature of real data distributions. We propose a learning framework to reduce this gap and achieve better normality representation.
  arXiv  Detail & Related papers  (2023-09-18T02:36:19Z)
• A Study of Representational Properties of Unsupervised Anomaly Detection in Brain MRI [1.376408511310322]
  Unsupervised methods for anomaly detection offer a way to observe properties related to factorization. We study four existing modeling methods, and report our empirical observations using simple data science tools. Our study indicates that anomaly detection algorithms that exhibit factorization related properties are well capacitated with delineatory capabilities.
  arXiv  Detail & Related papers  (2022-11-28T16:38:34Z)
• SLA$^2$P: Self-supervised Anomaly Detection with Adversarial Perturbation [77.71161225100927]
  Anomaly detection is a fundamental yet challenging problem in machine learning. We propose a novel and powerful framework, dubbed as SLA$2$P, for unsupervised anomaly detection.
  arXiv  Detail & Related papers  (2021-11-25T03:53:43Z)
• Understanding the Effect of Bias in Deep Anomaly Detection [15.83398707988473]
  Anomaly detection presents a unique challenge in machine learning, due to the scarcity of labeled anomaly data. Recent work attempts to mitigate such problems by augmenting training of deep anomaly detection models with additional labeled anomaly samples. In this paper, we aim to understand the effect of a biased anomaly set on anomaly detection.
  arXiv  Detail & Related papers  (2021-05-16T03:55:02Z)
• Deconfounded Score Method: Scoring DAGs with Dense Unobserved Confounding [101.35070661471124]
  We show that unobserved confounding leaves a characteristic footprint in the observed data distribution that allows for disentangling spurious and causal effects. We propose an adjusted score-based causal discovery algorithm that may be implemented with general-purpose solvers and scales to high-dimensional problems.
  arXiv  Detail & Related papers  (2021-03-28T11:07:59Z)
• On Disentangled Representations Learned From Correlated Data [59.41587388303554]
  We bridge the gap to real-world scenarios by analyzing the behavior of the most prominent disentanglement approaches on correlated data. We show that systematically induced correlations in the dataset are being learned and reflected in the latent representations. We also demonstrate how to resolve these latent correlations, either using weak supervision during training or by post-hoc correcting a pre-trained model with a small number of labels.
  arXiv  Detail & Related papers  (2020-06-14T12:47:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.