Large-Scale Evaluation of Open-Set Image Classification Techniques

• URL: http://arxiv.org/abs/2406.09112v1
• Date: Thu, 13 Jun 2024 13:43:01 GMT
• Title: Large-Scale Evaluation of Open-Set Image Classification Techniques
• Authors: Halil Bisgin, Andres Palechor, Mike Suter, Manuel Günther,
• Abstract summary: Open-Set Classification (OSC) algorithms aim to maximize both closed and open-set recognition capabilities. Recent studies showed the utility of such algorithms on small-scale data sets, but limited experimentation makes it difficult to assess their performances in real-world problems.
• Score: 1.1249583407496218
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The goal for classification is to correctly assign labels to unseen samples. However, most methods misclassify samples with unseen labels and assign them to one of the known classes. Open-Set Classification (OSC) algorithms aim to maximize both closed and open-set recognition capabilities. Recent studies showed the utility of such algorithms on small-scale data sets, but limited experimentation makes it difficult to assess their performances in real-world problems. Here, we provide a comprehensive comparison of various OSC algorithms, including training-based (SoftMax, Garbage, EOS) and post-processing methods (Maximum SoftMax Scores, Maximum Logit Scores, OpenMax, EVM, PROSER), the latter are applied on features from the former. We perform our evaluation on three large-scale protocols that mimic real-world challenges, where we train on known and negative open-set samples, and test on known and unknown instances. Our results show that EOS helps to improve performance of almost all post-processing algorithms. Particularly, OpenMax and PROSER are able to exploit better-trained networks, demonstrating the utility of hybrid models. However, while most algorithms work well on negative test samples -- samples of open-set classes seen during training -- they tend to perform poorly when tested on samples of previously unseen unknown classes, especially in challenging conditions.

Related papers

• Adapting Vision-Language Models to Open Classes via Test-Time Prompt Tuning [50.26965628047682]
  Adapting pre-trained models to open classes is a challenging problem in machine learning. In this paper, we consider combining the advantages of both and come up with a test-time prompt tuning approach. Our proposed method outperforms all comparison methods on average considering both base and new classes.
  arXiv  Detail & Related papers  (2024-08-29T12:34:01Z)
• Informed Decision-Making through Advancements in Open Set Recognition and Unknown Sample Detection [0.0]
  Open set recognition (OSR) aims to bring classification tasks in a situation that is more like reality. This study provides an algorithm exploring a new representation of feature space to improve classification in OSR tasks.
  arXiv  Detail & Related papers  (2024-05-09T15:15:34Z)
• Multiclass Learning from Noisy Labels for Non-decomposable Performance Measures [15.358504449550013]
  We design algorithms to learn from noisy labels for two broad classes of non-decomposable performance measures. In both cases, we develop noise-corrected versions of the algorithms under the widely studied class-conditional noise models. Our experiments demonstrate the effectiveness of our algorithms in handling label noise.
  arXiv  Detail & Related papers  (2024-02-01T23:03:53Z)
• Open-Set Facial Expression Recognition [42.62439125553367]
  Facial expression recognition (FER) models are typically trained on datasets with a fixed number of seven basic classes. Recent research works point out that there are far more expressions than the basic ones. We propose the open-set FER task for the first time.
  arXiv  Detail & Related papers  (2024-01-23T05:57:50Z)
• OpenAUC: Towards AUC-Oriented Open-Set Recognition [151.5072746015253]
  Traditional machine learning follows a close-set assumption that the training and test set share the same label space. Open-Set Recognition (OSR) aims to make correct predictions on both close-set samples and open-set samples. To fix these issues, we propose a novel metric named OpenAUC.
  arXiv  Detail & Related papers  (2022-10-22T08:54:15Z)
• Large-Scale Open-Set Classification Protocols for ImageNet [0.0]
  Open-Set Classification (OSC) intends to adapt closed-set classification models to real-world scenarios. We propose three open-set protocols that provide rich datasets of natural images with different levels of similarity between known and unknown classes. We propose a new validation metric that can be employed to assess whether the training of deep learning models addresses both the classification of known samples and the rejection of unknown samples.
  arXiv  Detail & Related papers  (2022-10-13T07:01:34Z)
• Towards Automated Imbalanced Learning with Deep Hierarchical Reinforcement Learning [57.163525407022966]
  Imbalanced learning is a fundamental challenge in data mining, where there is a disproportionate ratio of training samples in each class. Over-sampling is an effective technique to tackle imbalanced learning through generating synthetic samples for the minority class. We propose AutoSMOTE, an automated over-sampling algorithm that can jointly optimize different levels of decisions.
  arXiv  Detail & Related papers  (2022-08-26T04:28:01Z)
• Learning with Neighbor Consistency for Noisy Labels [69.83857578836769]
  We present a method for learning from noisy labels that leverages similarities between training examples in feature space. We evaluate our method on datasets evaluating both synthetic (CIFAR-10, CIFAR-100) and realistic (mini-WebVision, Clothing1M, mini-ImageNet-Red) noise.
  arXiv  Detail & Related papers  (2022-02-04T15:46:27Z)
• Reconstruction guided Meta-learning for Few Shot Open Set Recognition [31.49168444631114]
  We propose Reconstructing Exemplar-based Few-shot Open-set ClaSsifier (ReFOCS) By using a novel exemplar reconstruction-based meta-learning strategy ReFOCS streamlines FSOSR. We show ReFOCS to outperform multiple state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-07-31T23:23:35Z)
• Adaptive Sampling for Best Policy Identification in Markov Decision Processes [79.4957965474334]
  We investigate the problem of best-policy identification in discounted Markov Decision (MDPs) when the learner has access to a generative model. The advantages of state-of-the-art algorithms are discussed and illustrated.
  arXiv  Detail & Related papers  (2020-09-28T15:22:24Z)
• Few-Shot Open-Set Recognition using Meta-Learning [72.15940446408824]
  The problem of open-set recognition is considered. A new oPen sEt mEta LEaRning (PEELER) algorithm is introduced.
  arXiv  Detail & Related papers  (2020-05-27T23:49:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.