A View on Out-of-Distribution Identification from a Statistical Testing Theory Perspective

• URL: http://arxiv.org/abs/2405.03052v3
• Date: Fri, 10 May 2024 14:09:38 GMT
• Title: A View on Out-of-Distribution Identification from a Statistical Testing Theory Perspective
• Authors: Alberto Caron, Chris Hicks, Vasilios Mavroudis,
• Abstract summary: We study the problem of efficiently detecting Out-of-Distribution (OOD) samples at test time in supervised and unsupervised learning contexts. We re-formulate the OOD problem under the lenses of statistical testing and then discuss conditions that render the OOD problem identifiable in statistical terms.
• Score: 0.24578723416255752
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study the problem of efficiently detecting Out-of-Distribution (OOD) samples at test time in supervised and unsupervised learning contexts. While ML models are typically trained under the assumption that training and test data stem from the same distribution, this is often not the case in realistic settings, thus reliably detecting distribution shifts is crucial at deployment. We re-formulate the OOD problem under the lenses of statistical testing and then discuss conditions that render the OOD problem identifiable in statistical terms. Building on this framework, we study convergence guarantees of an OOD test based on the Wasserstein distance, and provide a simple empirical evaluation.

Related papers

• Rethinking Out-of-Distribution Detection on Imbalanced Data Distribution [38.844580833635725]
  We present a training-time regularization technique to mitigate the bias and boost imbalanced OOD detectors across architecture designs. Our method translates into consistent improvements on the representative CIFAR10-LT, CIFAR100-LT, and ImageNet-LT benchmarks.
  arXiv  Detail & Related papers  (2024-07-23T12:28:59Z)
• Detecting Out-of-Distribution Samples via Conditional Distribution Entropy with Optimal Transport [20.421338676377587]
  We argue that empirical probability distributions that incorporate geometric information from both training samples and test inputs can be highly beneficial for OOD detection. Within the framework of optimal transport, we propose a novel score function known as the emphconditional distribution entropy to quantify the uncertainty of a test input being an OOD sample.
  arXiv  Detail & Related papers  (2024-01-22T07:07:32Z)
• Model-free Test Time Adaptation for Out-Of-Distribution Detection [62.49795078366206]
  We propose a Non-Parametric Test Time textbfAdaptation framework for textbfDistribution textbfDetection (abbr) abbr utilizes online test samples for model adaptation during testing, enhancing adaptability to changing data distributions. We demonstrate the effectiveness of abbr through comprehensive experiments on multiple OOD detection benchmarks.
  arXiv  Detail & Related papers  (2023-11-28T02:00:47Z)
• Distilling the Unknown to Unveil Certainty [66.29929319664167]
  Out-of-distribution (OOD) detection is essential in identifying test samples that deviate from the in-distribution (ID) data upon which a standard network is trained. This paper introduces OOD knowledge distillation, a pioneering learning framework applicable whether or not training ID data is available.
  arXiv  Detail & Related papers  (2023-11-14T08:05:02Z)
• Plugin estimators for selective classification with out-of-distribution detection [67.28226919253214]
  Real-world classifiers can benefit from abstaining from predicting on samples where they have low confidence. These settings have been the subject of extensive but disjoint study in the selective classification (SC) and out-of-distribution (OOD) detection literature. Recent work on selective classification with OOD detection has argued for the unified study of these problems. We propose new plugin estimators for SCOD that are theoretically grounded, effective, and generalise existing approaches.
  arXiv  Detail & Related papers  (2023-01-29T07:45:17Z)
• Pseudo-OOD training for robust language models [78.15712542481859]
  OOD detection is a key component of a reliable machine-learning model for any industry-scale application. We propose POORE - POsthoc pseudo-Ood REgularization, that generates pseudo-OOD samples using in-distribution (IND) data. We extensively evaluate our framework on three real-world dialogue systems, achieving new state-of-the-art in OOD detection.
  arXiv  Detail & Related papers  (2022-10-17T14:32:02Z)
• Out-of-Distribution Detection with Hilbert-Schmidt Independence Optimization [114.43504951058796]
  Outlier detection tasks have been playing a critical role in AI safety. Deep neural network classifiers usually tend to incorrectly classify out-of-distribution (OOD) inputs into in-distribution classes with high confidence. We propose an alternative probabilistic paradigm that is both practically useful and theoretically viable for the OOD detection tasks.
  arXiv  Detail & Related papers  (2022-09-26T15:59:55Z)
• Multiple Testing Framework for Out-of-Distribution Detection [27.248375922343616]
  We study the problem of Out-of-Distribution (OOD) detection, that is, detecting whether a learning algorithm's output can be trusted at inference time. We propose a definition for the notion of OOD that includes both the input distribution and the learning algorithm, which provides insights for the construction of powerful tests for OOD detection.
  arXiv  Detail & Related papers  (2022-06-20T00:56:01Z)
• Understanding and Testing Generalization of Deep Networks on Out-of-Distribution Data [30.471871571256198]
  Deep network models perform excellently on In-Distribution data, but can significantly fail on Out-Of-Distribution data. This study is devoted to analyzing the problem of experimental ID test and designing OOD test paradigm.
  arXiv  Detail & Related papers  (2021-11-17T15:29:07Z)
• Statistical Testing for Efficient Out of Distribution Detection in Deep Neural Networks [26.0303701309125]
  This paper frames the Out Of Distribution (OOD) detection problem in Deep Neural Networks as a statistical hypothesis testing problem. We build on this framework to suggest a novel OOD procedure based on low-order statistics. Our method achieves comparable or better than state-of-the-art results on well-accepted OOD benchmarks without retraining the network parameters.
  arXiv  Detail & Related papers  (2021-02-25T16:14:47Z)
• Learn what you can't learn: Regularized Ensembles for Transductive Out-of-distribution Detection [76.39067237772286]
  We show that current out-of-distribution (OOD) detection algorithms for neural networks produce unsatisfactory results in a variety of OOD detection scenarios. This paper studies how such "hard" OOD scenarios can benefit from adjusting the detection method after observing a batch of the test data. We propose a novel method that uses an artificial labeling scheme for the test data and regularization to obtain ensembles of models that produce contradictory predictions only on the OOD samples in a test batch.
  arXiv  Detail & Related papers  (2020-12-10T16:55:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.