Learn what you can't learn: Regularized Ensembles for Transductive Out-of-distribution Detection

• URL: http://arxiv.org/abs/2012.05825v1
• Date: Thu, 10 Dec 2020 16:55:13 GMT
• Title: Learn what you can't learn: Regularized Ensembles for Transductive Out-of-distribution Detection
• Authors: Alexandru \c{T}ifrea, Eric Stavarache, Fanny Yang
• Abstract summary: We show that current out-of-distribution (OOD) detection algorithms for neural networks produce unsatisfactory results in a variety of OOD detection scenarios. This paper studies how such "hard" OOD scenarios can benefit from adjusting the detection method after observing a batch of the test data. We propose a novel method that uses an artificial labeling scheme for the test data and regularization to obtain ensembles of models that produce contradictory predictions only on the OOD samples in a test batch.
• Score: 76.39067237772286
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Machine learning models are often used in practice if they achieve good generalization results on in-distribution (ID) holdout data. When employed in the wild, they should also be able to detect samples they cannot predict well. We show that current out-of-distribution (OOD) detection algorithms for neural networks produce unsatisfactory results in a variety of OOD detection scenarios, e.g. when OOD data consists of unseen classes or corrupted measurements. This paper studies how such "hard" OOD scenarios can benefit from adjusting the detection method after observing a batch of the test data. This transductive setting is relevant when the advantage of even a slightly delayed OOD detection outweighs the financial cost for additional tuning. We propose a novel method that uses an artificial labeling scheme for the test data and regularization to obtain ensembles of models that produce contradictory predictions only on the OOD samples in a test batch. We show via comprehensive experiments that our approach is indeed able to significantly outperform both inductive and transductive baselines on difficult OOD detection scenarios, such as unseen classes on CIFAR-10/CIFAR-100, severe corruptions(CIFAR-C), and strong covariate shift (ImageNet vs ObjectNet).

Related papers

• Detecting Out-of-Distribution Samples via Conditional Distribution Entropy with Optimal Transport [20.421338676377587]
  We argue that empirical probability distributions that incorporate geometric information from both training samples and test inputs can be highly beneficial for OOD detection. Within the framework of optimal transport, we propose a novel score function known as the emphconditional distribution entropy to quantify the uncertainty of a test input being an OOD sample.
  arXiv  Detail & Related papers  (2024-01-22T07:07:32Z)
• Model-free Test Time Adaptation for Out-Of-Distribution Detection [62.49795078366206]
  We propose a Non-Parametric Test Time textbfAdaptation framework for textbfDistribution textbfDetection (abbr) abbr utilizes online test samples for model adaptation during testing, enhancing adaptability to changing data distributions. We demonstrate the effectiveness of abbr through comprehensive experiments on multiple OOD detection benchmarks.
  arXiv  Detail & Related papers  (2023-11-28T02:00:47Z)
• Out-of-distribution Detection Learning with Unreliable Out-of-distribution Sources [73.28967478098107]
  Out-of-distribution (OOD) detection discerns OOD data where the predictor cannot make valid predictions as in-distribution (ID) data. It is typically hard to collect real out-of-distribution (OOD) data for training a predictor capable of discerning OOD patterns. We propose a data generation-based learning method named Auxiliary Task-based OOD Learning (ATOL) that can relieve the mistaken OOD generation.
  arXiv  Detail & Related papers  (2023-11-06T16:26:52Z)
• AUTO: Adaptive Outlier Optimization for Online Test-Time OOD Detection [81.49353397201887]
  Out-of-distribution (OOD) detection is crucial to deploying machine learning models in open-world applications. We introduce a novel paradigm called test-time OOD detection, which utilizes unlabeled online data directly at test time to improve OOD detection performance. We propose adaptive outlier optimization (AUTO), which consists of an in-out-aware filter, an ID memory bank, and a semantically-consistent objective.
  arXiv  Detail & Related papers  (2023-03-22T02:28:54Z)
• Augmenting Softmax Information for Selective Classification with Out-of-Distribution Data [7.221206118679026]
  We show that existing post-hoc methods perform quite differently compared to when evaluated only on OOD detection. We propose a novel method for SCOD, Softmax Information Retaining Combination (SIRC), that augments softmax-based confidence scores with feature-agnostic information. Experiments on a wide variety of ImageNet-scale datasets and convolutional neural network architectures show that SIRC is able to consistently match or outperform the baseline for SCOD.
  arXiv  Detail & Related papers  (2022-07-15T14:39:57Z)
• Provably Robust Detection of Out-of-distribution Data (almost) for free [124.14121487542613]
  Deep neural networks are known to produce highly overconfident predictions on out-of-distribution (OOD) data. In this paper we propose a novel method where from first principles we combine a certifiable OOD detector with a standard classifier into an OOD aware classifier. In this way we achieve the best of two worlds: certifiably adversarially robust OOD detection, even for OOD samples close to the in-distribution, without loss in prediction accuracy and close to state-of-the-art OOD detection performance for non-manipulated OOD data.
  arXiv  Detail & Related papers  (2021-06-08T11:40:49Z)
• Statistical Testing for Efficient Out of Distribution Detection in Deep Neural Networks [26.0303701309125]
  This paper frames the Out Of Distribution (OOD) detection problem in Deep Neural Networks as a statistical hypothesis testing problem. We build on this framework to suggest a novel OOD procedure based on low-order statistics. Our method achieves comparable or better than state-of-the-art results on well-accepted OOD benchmarks without retraining the network parameters.
  arXiv  Detail & Related papers  (2021-02-25T16:14:47Z)
• Robust Out-of-distribution Detection for Neural Networks [51.19164318924997]
  We show that existing detection mechanisms can be extremely brittle when evaluating on in-distribution and OOD inputs. We propose an effective algorithm called ALOE, which performs robust training by exposing the model to both adversarially crafted inlier and outlier examples.
  arXiv  Detail & Related papers  (2020-03-21T17:46:28Z)
• Detecting Out-of-Distribution Examples with In-distribution Examples and Gram Matrices [8.611328447624679]
  Deep neural networks yield confident, incorrect predictions when presented with Out-of-Distribution examples. In this paper, we propose to detect OOD examples by identifying inconsistencies between activity patterns and class predicted. We find that characterizing activity patterns by Gram matrices and identifying anomalies in gram matrix values can yield high OOD detection rates.
  arXiv  Detail & Related papers  (2019-12-28T19:44:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.