Towards Realistic Out-of-Distribution Detection: A Novel Evaluation Framework for Improving Generalization in OOD Detection

• URL: http://arxiv.org/abs/2211.10892v2
• Date: Thu, 31 Aug 2023 12:09:45 GMT
• Title: Towards Realistic Out-of-Distribution Detection: A Novel Evaluation Framework for Improving Generalization in OOD Detection
• Authors: Vahid Reza Khazaie and Anthony Wong and Mohammad Sabokrou
• Abstract summary: This paper presents a novel evaluation framework for Out-of-Distribution (OOD) detection. It aims to assess the performance of machine learning models in more realistic settings.
• Score: 14.541761912174799
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper presents a novel evaluation framework for Out-of-Distribution (OOD) detection that aims to assess the performance of machine learning models in more realistic settings. We observed that the real-world requirements for testing OOD detection methods are not satisfied by the current testing protocols. They usually encourage methods to have a strong bias towards a low level of diversity in normal data. To address this limitation, we propose new OOD test datasets (CIFAR-10-R, CIFAR-100-R, and ImageNet-30-R) that can allow researchers to benchmark OOD detection performance under realistic distribution shifts. Additionally, we introduce a Generalizability Score (GS) to measure the generalization ability of a model during OOD detection. Our experiments demonstrate that improving the performance on existing benchmark datasets does not necessarily improve the usability of OOD detection models in real-world scenarios. While leveraging deep pre-trained features has been identified as a promising avenue for OOD detection research, our experiments show that state-of-the-art pre-trained models tested on our proposed datasets suffer a significant drop in performance. To address this issue, we propose a post-processing stage for adapting pre-trained features under these distribution shifts before calculating the OOD scores, which significantly enhances the performance of state-of-the-art pre-trained models on our benchmarks.

Related papers

• Model-free Test Time Adaptation for Out-Of-Distribution Detection [62.49795078366206]
  We propose a Non-Parametric Test Time textbfAdaptation framework for textbfDistribution textbfDetection (abbr) abbr utilizes online test samples for model adaptation during testing, enhancing adaptability to changing data distributions. We demonstrate the effectiveness of abbr through comprehensive experiments on multiple OOD detection benchmarks.
  arXiv  Detail & Related papers  (2023-11-28T02:00:47Z)
• Can Pre-trained Networks Detect Familiar Out-of-Distribution Data? [37.36999826208225]
  We study the effect of PT-OOD on the OOD detection performance of pre-trained networks. We find that the low linear separability of PT-OOD in the feature space heavily degrades the PT-OOD detection performance. We propose a unique solution to large-scale pre-trained models: Leveraging powerful instance-by-instance discriminative representations of pre-trained models.
  arXiv  Detail & Related papers  (2023-10-02T02:01:00Z)
• AUTO: Adaptive Outlier Optimization for Online Test-Time OOD Detection [81.49353397201887]
  Out-of-distribution (OOD) detection is crucial to deploying machine learning models in open-world applications. We introduce a novel paradigm called test-time OOD detection, which utilizes unlabeled online data directly at test time to improve OOD detection performance. We propose adaptive outlier optimization (AUTO), which consists of an in-out-aware filter, an ID memory bank, and a semantically-consistent objective.
  arXiv  Detail & Related papers  (2023-03-22T02:28:54Z)
• Unsupervised Evaluation of Out-of-distribution Detection: A Data-centric Perspective [55.45202687256175]
  Out-of-distribution (OOD) detection methods assume that they have test ground truths, i.e., whether individual test samples are in-distribution (IND) or OOD. In this paper, we are the first to introduce the unsupervised evaluation problem in OOD detection. We propose three methods to compute Gscore as an unsupervised indicator of OOD detection performance.
  arXiv  Detail & Related papers  (2023-02-16T13:34:35Z)
• Rethinking Out-of-distribution (OOD) Detection: Masked Image Modeling is All You Need [52.88953913542445]
  We find surprisingly that simply using reconstruction-based methods could boost the performance of OOD detection significantly. We take Masked Image Modeling as a pretext task for our OOD detection framework (MOOD)
  arXiv  Detail & Related papers  (2023-02-06T08:24:41Z)
• Pseudo-OOD training for robust language models [78.15712542481859]
  OOD detection is a key component of a reliable machine-learning model for any industry-scale application. We propose POORE - POsthoc pseudo-Ood REgularization, that generates pseudo-OOD samples using in-distribution (IND) data. We extensively evaluate our framework on three real-world dialogue systems, achieving new state-of-the-art in OOD detection.
  arXiv  Detail & Related papers  (2022-10-17T14:32:02Z)
• How Useful are Gradients for OOD Detection Really? [5.459639971144757]
  Out of distribution (OOD) detection is a critical challenge in deploying highly performant machine learning models in real-life applications. We provide an in-depth analysis and comparison of gradient based methods for OOD detection. We propose a general, non-gradient based method of OOD detection which improves over previous baselines in both performance and computational efficiency.
  arXiv  Detail & Related papers  (2022-05-20T21:10:05Z)
• RODD: A Self-Supervised Approach for Robust Out-of-Distribution Detection [12.341250124228859]
  We propose a simple yet effective generalized OOD detection method independent of out-of-distribution datasets. Our approach relies on self-supervised feature learning of the training samples, where the embeddings lie on a compact low-dimensional space. We empirically show that a pre-trained model with self-supervised contrastive learning yields a better model for uni-dimensional feature learning in the latent space.
  arXiv  Detail & Related papers  (2022-04-06T03:05:58Z)
• ReAct: Out-of-distribution Detection With Rectified Activations [20.792140933660075]
  Out-of-distribution (OOD) detection has received much attention lately due to its practical importance. One of the primary challenges is that models often produce highly confident predictions on OOD data. We propose ReAct--a simple and effective technique for reducing model overconfidence on OOD data.
  arXiv  Detail & Related papers  (2021-11-24T21:02:07Z)
• Robust Out-of-distribution Detection for Neural Networks [51.19164318924997]
  We show that existing detection mechanisms can be extremely brittle when evaluating on in-distribution and OOD inputs. We propose an effective algorithm called ALOE, which performs robust training by exposing the model to both adversarially crafted inlier and outlier examples.
  arXiv  Detail & Related papers  (2020-03-21T17:46:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.