An Insight on Evaluation Metrics Under the Imbalanced Case of Anomaly Detection

• URL: http://arxiv.org/abs/2607.22286v1
• Date: Fri, 24 Jul 2026 13:25:06 GMT
• Title: An Insight on Evaluation Metrics Under the Imbalanced Case of Anomaly Detection
• Abstract summary: We focus on the study of metric landscapes that relate metric values to true positive and true negative rates.<n>Our analysis offers practical guidance for interpreting and comparing anomaly detection results across datasets with different imbalance ratios.
• Score: 15.306867582514686
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Anomaly detection is inherently characterised by severe class imbalance, making the interpretation of evaluation metrics challenging. Although metrics such as AUROC, AUPR, F1-score, and MCC are widely used, their values convey different meanings depending on the anomaly ratio. In this work, we analyse the behaviour of those four common anomaly detection metrics under varying levels of imbalance. We focus on the study of metric landscapes, visualisations that relate metric values to true positive and true negative rates, providing an intuitive view of metric preferences and stability. Our analysis offers practical guidance for interpreting and comparing anomaly detection results across datasets with different imbalance ratios.

Related papers

• When Fairness Metrics Disagree: Evaluating the Reliability of Demographic Fairness Assessment in Machine Learning [0.0]
  We investigate the consistency of fairness evaluation by conducting a systematic multi-metric analysis of demographic bias in machine learning models.<n>Our results demonstrate that fairness assessments can vary significantly depending on the choice of metrics, leading to contradictory conclusions regarding model bias.<n>These findings highlight a critical limitation in current fairness evaluation practices and suggest that single-metric reporting is insufficient for reliable bias assessment.
  arXiv  Detail & Related papers  (2026-04-16T14:07:37Z)
• A Theoretical Framework for Statistical Evaluability of Generative Models [57.9316356505791]
  We introduce a theoretical framework for evaluating generative models and establish evaluability results for commonly used metrics.<n>We study two categories of metrics: test-based metrics, including integral probability metrics (IPMs) and Rényi divergences.<n>We show that IPMs with respect to any bounded test class can be evaluated from finite samples up to multiplicative and additive approximation errors.<n>In contrast, Rényi and KL divergences are not evaluable from finite samples, as their values can be critically determined by rare events.
  arXiv  Detail & Related papers  (2026-04-07T01:53:59Z)
• A Problem-Oriented Taxonomy of Evaluation Metrics for Time Series Anomaly Detection [11.114573387946498]
  Time series anomaly detection is widely used in IoT and cyber-physical systems.<n>This study introduces a problem-oriented framework that reinterprets existing metrics based on the specific evaluation challenges they are designed to address.
  arXiv  Detail & Related papers  (2025-11-24T04:09:04Z)
• Performance of the empirical median for location estimation in heteroscedastic settings [0.0]
  We consider independent symmetric real-valued random variables that share a common but unknown location parameter.<n>We derive matching non-asymptotic upper and lower bounds on its estimation error.
  arXiv  Detail & Related papers  (2025-01-28T13:57:54Z)
• Exploring the Impact of Outlier Variability on Anomaly Detection Evaluation Metrics [4.943054375935879]
  This study focuses on examining the behaviors of three widely used anomaly detection metrics under different conditions. We present findings that challenge the conventional understanding of these metrics and reveal nuanced behaviors under varying conditions. The results of our study contribute to a more refined understanding of metric selection and interpretation in anomaly detection, offering valuable insights for both researchers and practitioners in the field.
  arXiv  Detail & Related papers  (2024-09-24T11:39:09Z)
• On Pixel-level Performance Assessment in Anomaly Detection [87.7131059062292]
  Anomaly detection methods have demonstrated remarkable success across various applications. However, assessing their performance, particularly at the pixel-level, presents a complex challenge. In this paper, we dissect the intricacies of this challenge, underscored by visual evidence and statistical analysis.
  arXiv  Detail & Related papers  (2023-10-25T08:02:27Z)
• BLEURT Has Universal Translations: An Analysis of Automatic Metrics by Minimum Risk Training [64.37683359609308]
  In this study, we analyze various mainstream and cutting-edge automatic metrics from the perspective of their guidance for training machine translation systems. We find that certain metrics exhibit robustness defects, such as the presence of universal adversarial translations in BLEURT and BARTScore. In-depth analysis suggests two main causes of these robustness deficits: distribution biases in the training datasets, and the tendency of the metric paradigm.
  arXiv  Detail & Related papers  (2023-07-06T16:59:30Z)
• Measuring Class-Imbalance Sensitivity of Deterministic Performance Evaluation Metrics [0.8376091455761261]
  We introduce an intuitive evaluation framework that quantifies metrics' sensitivity to the class imbalance. We reveal that there is a logarithmic behavior in metrics' sensitivity meaning that the higher imbalance ratios are associated with the lower sensitivity of metrics.
  arXiv  Detail & Related papers  (2022-06-20T19:56:33Z)
• A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methods [60.04142561088524]
  We find that the confidence intervals are rather wide, demonstrating high uncertainty in how reliable automatic metrics truly are. Although many metrics fail to show statistical improvements over ROUGE, two recent works, QAEval and BERTScore, do in some evaluation settings.
  arXiv  Detail & Related papers  (2021-03-31T18:28:14Z)
• On Disentangled Representations Learned From Correlated Data [59.41587388303554]
  We bridge the gap to real-world scenarios by analyzing the behavior of the most prominent disentanglement approaches on correlated data. We show that systematically induced correlations in the dataset are being learned and reflected in the latent representations. We also demonstrate how to resolve these latent correlations, either using weak supervision during training or by post-hoc correcting a pre-trained model with a small number of labels.
  arXiv  Detail & Related papers  (2020-06-14T12:47:34Z)
• Learning from Aggregate Observations [82.44304647051243]
  We study the problem of learning from aggregate observations where supervision signals are given to sets of instances. We present a general probabilistic framework that accommodates a variety of aggregate observations. Simple maximum likelihood solutions can be applied to various differentiable models.
  arXiv  Detail & Related papers  (2020-04-14T06:18:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.