Model-free Test Time Adaptation for Out-Of-Distribution Detection
- URL: http://arxiv.org/abs/2311.16420v1
- Date: Tue, 28 Nov 2023 02:00:47 GMT
- Title: Model-free Test Time Adaptation for Out-Of-Distribution Detection
- Authors: YiFan Zhang, Xue Wang, Tian Zhou, Kun Yuan, Zhang Zhang, Liang Wang,
Rong Jin, Tieniu Tan
- Abstract summary: We propose a Non-Parametric Test Time textbfAdaptation framework for textbfDistribution textbfDetection (abbr)
abbr utilizes online test samples for model adaptation during testing, enhancing adaptability to changing data distributions.
We demonstrate the effectiveness of abbr through comprehensive experiments on multiple OOD detection benchmarks.
- Score: 62.49795078366206
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Out-of-distribution (OOD) detection is essential for the reliability of ML
models. Most existing methods for OOD detection learn a fixed decision
criterion from a given in-distribution dataset and apply it universally to
decide if a data point is OOD. Recent work~\cite{fang2022is} shows that given
only in-distribution data, it is impossible to reliably detect OOD data without
extra assumptions. Motivated by the theoretical result and recent exploration
of test-time adaptation methods, we propose a Non-Parametric Test Time
\textbf{Ada}ptation framework for \textbf{O}ut-Of-\textbf{D}istribution
\textbf{D}etection (\abbr). Unlike conventional methods, \abbr utilizes online
test samples for model adaptation during testing, enhancing adaptability to
changing data distributions. The framework incorporates detected OOD instances
into decision-making, reducing false positive rates, particularly when ID and
OOD distributions overlap significantly. We demonstrate the effectiveness of
\abbr through comprehensive experiments on multiple OOD detection benchmarks,
extensive empirical studies show that \abbr significantly improves the
performance of OOD detection over state-of-the-art methods. Specifically, \abbr
reduces the false positive rate (FPR95) by $23.23\%$ on the CIFAR-10 benchmarks
and $38\%$ on the ImageNet-1k benchmarks compared to the advanced methods.
Lastly, we theoretically verify the effectiveness of \abbr.
Related papers
- Detecting Out-of-Distribution Samples via Conditional Distribution
Entropy with Optimal Transport [20.421338676377587]
We argue that empirical probability distributions that incorporate geometric information from both training samples and test inputs can be highly beneficial for OOD detection.
Within the framework of optimal transport, we propose a novel score function known as the emphconditional distribution entropy to quantify the uncertainty of a test input being an OOD sample.
arXiv Detail & Related papers (2024-01-22T07:07:32Z) - Beyond AUROC & co. for evaluating out-of-distribution detection
performance [50.88341818412508]
Given their relevance for safe(r) AI, it is important to examine whether the basis for comparing OOD detection methods is consistent with practical needs.
We propose a new metric - Area Under the Threshold Curve (AUTC), which explicitly penalizes poor separation between ID and OOD samples.
arXiv Detail & Related papers (2023-06-26T12:51:32Z) - AUTO: Adaptive Outlier Optimization for Online Test-Time OOD Detection [81.49353397201887]
Out-of-distribution (OOD) detection is crucial to deploying machine learning models in open-world applications.
We introduce a novel paradigm called test-time OOD detection, which utilizes unlabeled online data directly at test time to improve OOD detection performance.
We propose adaptive outlier optimization (AUTO), which consists of an in-out-aware filter, an ID memory bank, and a semantically-consistent objective.
arXiv Detail & Related papers (2023-03-22T02:28:54Z) - Augmenting Softmax Information for Selective Classification with
Out-of-Distribution Data [7.221206118679026]
We show that existing post-hoc methods perform quite differently compared to when evaluated only on OOD detection.
We propose a novel method for SCOD, Softmax Information Retaining Combination (SIRC), that augments softmax-based confidence scores with feature-agnostic information.
Experiments on a wide variety of ImageNet-scale datasets and convolutional neural network architectures show that SIRC is able to consistently match or outperform the baseline for SCOD.
arXiv Detail & Related papers (2022-07-15T14:39:57Z) - Out-of-distribution Detection with Deep Nearest Neighbors [33.71627349163909]
Out-of-distribution (OOD) detection is a critical task for deploying machine learning models in the open world.
In this paper, we explore the efficacy of non-parametric nearest-neighbor distance for OOD detection.
We demonstrate the effectiveness of nearest-neighbor-based OOD detection on several benchmarks and establish superior performance.
arXiv Detail & Related papers (2022-04-13T16:45:21Z) - Label Smoothed Embedding Hypothesis for Out-of-Distribution Detection [72.35532598131176]
We propose an unsupervised method to detect OOD samples using a $k$-NN density estimate.
We leverage a recent insight about label smoothing, which we call the emphLabel Smoothed Embedding Hypothesis
We show that our proposal outperforms many OOD baselines and also provide new finite-sample high-probability statistical results.
arXiv Detail & Related papers (2021-02-09T21:04:44Z) - Learn what you can't learn: Regularized Ensembles for Transductive
Out-of-distribution Detection [76.39067237772286]
We show that current out-of-distribution (OOD) detection algorithms for neural networks produce unsatisfactory results in a variety of OOD detection scenarios.
This paper studies how such "hard" OOD scenarios can benefit from adjusting the detection method after observing a batch of the test data.
We propose a novel method that uses an artificial labeling scheme for the test data and regularization to obtain ensembles of models that produce contradictory predictions only on the OOD samples in a test batch.
arXiv Detail & Related papers (2020-12-10T16:55:13Z) - Robust Out-of-distribution Detection for Neural Networks [51.19164318924997]
We show that existing detection mechanisms can be extremely brittle when evaluating on in-distribution and OOD inputs.
We propose an effective algorithm called ALOE, which performs robust training by exposing the model to both adversarially crafted inlier and outlier examples.
arXiv Detail & Related papers (2020-03-21T17:46:28Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.