Robust performance metrics for imbalanced classification problems

• URL: http://arxiv.org/abs/2404.07661v1
• Date: Thu, 11 Apr 2024 11:50:05 GMT
• Title: Robust performance metrics for imbalanced classification problems
• Authors: Hajo Holzmann, Bernhard Klar,
• Abstract summary: We show that established performance metrics in binary classification, such as the F-score, are not robust to class imbalance. We introduce robust modifications of the F-score and the MCC for which, even in strongly imbalanced settings, the TPR is bounded away from $0$.
• Score: 2.07180164747172
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We show that established performance metrics in binary classification, such as the F-score, the Jaccard similarity coefficient or Matthews' correlation coefficient (MCC), are not robust to class imbalance in the sense that if the proportion of the minority class tends to $0$, the true positive rate (TPR) of the Bayes classifier under these metrics tends to $0$ as well. Thus, in imbalanced classification problems, these metrics favour classifiers which ignore the minority class. To alleviate this issue we introduce robust modifications of the F-score and the MCC for which, even in strongly imbalanced settings, the TPR is bounded away from $0$. We numerically illustrate the behaviour of the various performance metrics in simulations as well as on a credit default data set. We also discuss connections to the ROC and precision-recall curves and give recommendations on how to combine their usage with performance metrics.

Related papers

• Noisy Correspondence Learning with Self-Reinforcing Errors Mitigation [63.180725016463974]
  Cross-modal retrieval relies on well-matched large-scale datasets that are laborious in practice. We introduce a novel noisy correspondence learning framework, namely textbfSelf-textbfReinforcing textbfErrors textbfMitigation (SREM)
  arXiv  Detail & Related papers  (2023-12-27T09:03:43Z)
• Balanced Classification: A Unified Framework for Long-Tailed Object Detection [74.94216414011326]
  Conventional detectors suffer from performance degradation when dealing with long-tailed data due to a classification bias towards the majority head categories. We introduce a unified framework called BAlanced CLassification (BACL), which enables adaptive rectification of inequalities caused by disparities in category distribution. BACL consistently achieves performance improvements across various datasets with different backbones and architectures.
  arXiv  Detail & Related papers  (2023-08-04T09:11:07Z)
• Improving Multi-class Classifier Using Likelihood Ratio Estimation with Regularization [0.0]
  The universal-set naive Bayes classifier (UNB)citeKomiya:13, defined using likelihood ratios (LRs) was proposed to address imbalanced classification problems. The LR estimator used in the UNB overestimates LRs for low-frequency data, degrading the classification performance. Our experiments with imbalanced data showed that our proposed classifier effectively adjusts the classification scores according to the class balance using regularization parameters.
  arXiv  Detail & Related papers  (2022-10-28T10:07:53Z)
• Never mind the metrics -- what about the uncertainty? Visualising confusion matrix metric distributions [6.566615606042994]
  This paper strives for a more balanced perspective on classifier performance metrics by highlighting their distributions under different models of uncertainty. We develop equations, animations and interactive visualisations of the contours of performance metrics within (and beyond) this ROC space. Our hope is that these insights and visualisations will raise greater awareness of the substantial uncertainty in performance metric estimates.
  arXiv  Detail & Related papers  (2022-06-05T11:54:59Z)
• PLM: Partial Label Masking for Imbalanced Multi-label Classification [59.68444804243782]
  Neural networks trained on real-world datasets with long-tailed label distributions are biased towards frequent classes and perform poorly on infrequent classes. We propose a method, Partial Label Masking (PLM), which utilizes this ratio during training. Our method achieves strong performance when compared to existing methods on both multi-label (MultiMNIST and MSCOCO) and single-label (imbalanced CIFAR-10 and CIFAR-100) image classification datasets.
  arXiv  Detail & Related papers  (2021-05-22T18:07:56Z)
• Deconfounding Scores: Feature Representations for Causal Effect Estimation with Weak Overlap [140.98628848491146]
  We introduce deconfounding scores, which induce better overlap without biasing the target of estimation. We show that deconfounding scores satisfy a zero-covariance condition that is identifiable in observed data. In particular, we show that this technique could be an attractive alternative to standard regularizations.
  arXiv  Detail & Related papers  (2021-04-12T18:50:11Z)
• Intra-Class Uncertainty Loss Function for Classification [6.523198497365588]
  intra-class uncertainty/variability is not considered, especially for datasets containing unbalanced classes. In our framework, the features extracted by deep networks of each class are characterized by independent Gaussian distribution. The proposed approach shows improved classification performance, through learning a better class representation.
  arXiv  Detail & Related papers  (2021-04-12T09:02:41Z)
• Label-Imbalanced and Group-Sensitive Classification under Overparameterization [32.923780772605596]
  Label-imbalanced and group-sensitive classification seeks to appropriately modify standard training algorithms to optimize relevant metrics. We show that a logit-adjusted loss modification to standard empirical risk minimization might be ineffective in general. We show that our results extend naturally to binary classification with sensitive groups, thus treating the two common types of imbalances (label/group) in a unifying way.
  arXiv  Detail & Related papers  (2021-03-02T08:09:43Z)
• Re-Assessing the "Classify and Count" Quantification Method [88.60021378715636]
  "Classify and Count" (CC) is often a biased estimator. Previous works have failed to use properly optimised versions of CC. We argue that, while still inferior to some cutting-edge methods, they deliver near-state-of-the-art accuracy.
  arXiv  Detail & Related papers  (2020-11-04T21:47:39Z)
• Pointwise Binary Classification with Pairwise Confidence Comparisons [97.79518780631457]
  We propose pairwise comparison (Pcomp) classification, where we have only pairs of unlabeled data that we know one is more likely to be positive than the other. We link Pcomp classification to noisy-label learning to develop a progressive URE and improve it by imposing consistency regularization.
  arXiv  Detail & Related papers  (2020-10-05T09:23:58Z)
• Classification Performance Metric for Imbalance Data Based on Recall and Selectivity Normalized in Class Labels [0.0]
  We introduce a new performance measure based on the harmonic mean of Recall and Selectivity normalized in class labels. This paper shows that the proposed performance measure has the right properties for the imbalanced dataset.
  arXiv  Detail & Related papers  (2020-06-23T20:38:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.