Taking Class Imbalance Into Account in Open Set Recognition Evaluation

• URL: http://arxiv.org/abs/2402.06331v1
• Date: Fri, 9 Feb 2024 11:15:49 GMT
• Title: Taking Class Imbalance Into Account in Open Set Recognition Evaluation
• Abstract summary: This work looks at the evaluation of methods for Open Set Recognition. As an outcome of problem analysis, we present a set of guidelines for evaluation of methods in this field.
• Score: 0.3626013617212667
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: In recent years Deep Neural Network-based systems are not only increasing in popularity but also receive growing user trust. However, due to the closed-world assumption of such systems, they cannot recognize samples from unknown classes and often induce an incorrect label with high confidence. Presented work looks at the evaluation of methods for Open Set Recognition, focusing on the impact of class imbalance, especially in the dichotomy between known and unknown samples. As an outcome of problem analysis, we present a set of guidelines for evaluation of methods in this field.

Related papers

• Evidence-Guided Unknown Rejection for High-Confidence Near-Known Unknowns [4.320698236637654]
  We show that open-set recognition systems face a neglected failure mode: high-confidence near-known unknowns.<n>We propose EGUR-A, which changes the decision from is this sample's score high enough?'' to does this predicted known class have sufficient evidence to accept this sample?''
  arXiv  Detail & Related papers  (2026-05-18T03:46:55Z)
• Known Meets Unknown: Mitigating Overconfidence in Open Set Recognition [4.377912830814393]
  Open Set Recognition (OSR) requires models to accurately classify known classes and to effectively reject unknown samples.<n>When unknown samples are semantically similar to known classes, inter-class overlap in the feature space often causes models to assign unjustifiably high confidence to them.<n>We propose a framework that explicitly mitigates overconfidence caused by inter-class overlap.
  arXiv  Detail & Related papers  (2025-11-15T09:56:44Z)
• Open Set Recognition for Random Forest [4.266270583680947]
  In real-world classification tasks, it is difficult to collect training examples that exhaust all possible classes. We propose a novel approach to enabling open-set recognition capability for random forest. The proposed method is validated on both synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2024-08-01T04:21:14Z)
• Revisiting Confidence Estimation: Towards Reliable Failure Prediction [53.79160907725975]
  We find a general, widely existing but actually-neglected phenomenon that most confidence estimation methods are harmful for detecting misclassification errors. We propose to enlarge the confidence gap by finding flat minima, which yields state-of-the-art failure prediction performance.
  arXiv  Detail & Related papers  (2024-03-05T11:44:14Z)
• Managing the unknown: a survey on Open Set Recognition and tangential areas [7.345136916791223]
  Open Set Recognition models are capable of detecting unknown classes from samples arriving during the testing phase, while maintaining a good level of performance in the classification of samples belonging to known classes. This review comprehensively overviews the recent literature related to Open Set Recognition, identifying common practices, limitations, and connections of this field with other machine learning research areas. Our work also uncovers open problems and suggests several research directions that may motivate and articulate future efforts towards more safe Artificial Intelligence methods.
  arXiv  Detail & Related papers  (2023-12-14T10:08:12Z)
• OpenMix: Exploring Outlier Samples for Misclassification Detection [37.43981354073841]
  We exploit the easily available outlier samples, i.e., unlabeled samples coming from non-target classes, for helping detect misclassification errors. We propose a novel method called OpenMix, which incorporates open-world knowledge by learning to reject uncertain pseudo-samples generated via outlier transformation.
  arXiv  Detail & Related papers  (2023-03-30T01:47:23Z)
• Uncertainty Estimation by Fisher Information-based Evidential Deep Learning [61.94125052118442]
  Uncertainty estimation is a key factor that makes deep learning reliable in practical applications. We propose a novel method, Fisher Information-based Evidential Deep Learning ($mathcalI$-EDL) In particular, we introduce Fisher Information Matrix (FIM) to measure the informativeness of evidence carried by each sample, according to which we can dynamically reweight the objective loss terms to make the network more focused on the representation learning of uncertain classes.
  arXiv  Detail & Related papers  (2023-03-03T16:12:59Z)
• Benchmarking common uncertainty estimation methods with histopathological images under domain shift and label noise [62.997667081978825]
  In high-risk environments, deep learning models need to be able to judge their uncertainty and reject inputs when there is a significant chance of misclassification. We conduct a rigorous evaluation of the most commonly used uncertainty and robustness methods for the classification of Whole Slide Images. We observe that ensembles of methods generally lead to better uncertainty estimates as well as an increased robustness towards domain shifts and label noise.
  arXiv  Detail & Related papers  (2023-01-03T11:34:36Z)
• What is Flagged in Uncertainty Quantification? Latent Density Models for Uncertainty Categorization [68.15353480798244]
  Uncertainty Quantification (UQ) is essential for creating trustworthy machine learning models. Recent years have seen a steep rise in UQ methods that can flag suspicious examples. We propose a framework for categorizing uncertain examples flagged by UQ methods in classification tasks.
  arXiv  Detail & Related papers  (2022-07-11T19:47:00Z)
• Open Set Recognition with Conditional Probabilistic Generative Models [51.40872765917125]
  We propose Conditional Probabilistic Generative Models (CPGM) for open set recognition. CPGM can detect unknown samples but also classify known classes by forcing different latent features to approximate conditional Gaussian distributions. Experiment results on multiple benchmark datasets reveal that the proposed method significantly outperforms the baselines.
  arXiv  Detail & Related papers  (2020-08-12T06:23:49Z)
• Domain Knowledge Alleviates Adversarial Attacks in Multi-Label Classifiers [34.526394646264734]
  Adversarial attacks on machine learning-based classifiers, along with defense mechanisms, have been widely studied. In this paper, we shift the attention to multi-label classification, where the availability of domain knowledge may offer a natural way to spot incoherent predictions. We explore this intuition in a framework in which first-order logic knowledge is converted into constraints and injected into a semi-supervised learning problem.
  arXiv  Detail & Related papers  (2020-06-06T10:24:54Z)
• Conditional Gaussian Distribution Learning for Open Set Recognition [10.90687687505665]
  We propose Conditional Gaussian Distribution Learning (CGDL) for open set recognition. In addition to detecting unknown samples, this method can also classify known samples by forcing different latent features to approximate different Gaussian models. Experiments on several standard image reveal that the proposed method significantly outperforms the baseline method and achieves new state-of-the-art results.
  arXiv  Detail & Related papers  (2020-03-19T14:32:08Z)
• Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning [70.72363097550483]
  In this study, we focus on in-domain uncertainty for image classification. To provide more insight in this study, we introduce the deep ensemble equivalent score (DEE)
  arXiv  Detail & Related papers  (2020-02-15T23:28:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.