Exploring Covariate and Concept Shift for Detection and Calibration of
Out-of-Distribution Data
- URL: http://arxiv.org/abs/2110.15231v1
- Date: Thu, 28 Oct 2021 15:42:55 GMT
- Title: Exploring Covariate and Concept Shift for Detection and Calibration of
Out-of-Distribution Data
- Authors: Junjiao Tian, Yen-Change Hsu, Yilin Shen, Hongxia Jin, Zsolt Kira
- Abstract summary: characterization reveals that sensitivity to each type of shift is important to the detection and confidence calibration of OOD data.
We propose a geometrically-inspired method to improve OOD detection under both shifts with only in-distribution data.
We are the first to propose a method that works well across both OOD detection and calibration and under different types of shifts.
- Score: 77.27338842609153
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Moving beyond testing on in-distribution data works on Out-of-Distribution
(OOD) detection have recently increased in popularity. A recent attempt to
categorize OOD data introduces the concept of near and far OOD detection.
Specifically, prior works define characteristics of OOD data in terms of
detection difficulty. We propose to characterize the spectrum of OOD data using
two types of distribution shifts: covariate shift and concept shift, where
covariate shift corresponds to change in style, e.g., noise, and concept shift
indicates a change in semantics. This characterization reveals that sensitivity
to each type of shift is important to the detection and confidence calibration
of OOD data. Consequently, we investigate score functions that capture
sensitivity to each type of dataset shift and methods that improve them. To
this end, we theoretically derive two score functions for OOD detection, the
covariate shift score and concept shift score, based on the decomposition of
KL-divergence for both scores, and propose a geometrically-inspired method
(Geometric ODIN) to improve OOD detection under both shifts with only
in-distribution data. Additionally, the proposed method naturally leads to an
expressive post-hoc calibration function which yields state-of-the-art
calibration performance on both in-distribution and out-of-distribution data.
We are the first to propose a method that works well across both OOD detection
and calibration and under different types of shifts. Specifically, we improve
the previous state-of-the-art OOD detection by relatively 7% AUROC on CIFAR100
vs. SVHN and achieve the best calibration performance of 0.084 Expected
Calibration Error on the corrupted CIFAR100C dataset. View project page at
https://sites.google.com/view/geometric-decomposition.
Related papers
- Rethinking the Evaluation of Out-of-Distribution Detection: A Sorites Paradox [70.57120710151105]
Most existing out-of-distribution (OOD) detection benchmarks classify samples with novel labels as the OOD data.
Some marginal OOD samples actually have close semantic contents to the in-distribution (ID) sample, which makes determining the OOD sample a Sorites Paradox.
We construct a benchmark named Incremental Shift OOD (IS-OOD) to address the issue.
arXiv Detail & Related papers (2024-06-14T09:27:56Z) - Model-free Test Time Adaptation for Out-Of-Distribution Detection [62.49795078366206]
We propose a Non-Parametric Test Time textbfAdaptation framework for textbfDistribution textbfDetection (abbr)
abbr utilizes online test samples for model adaptation during testing, enhancing adaptability to changing data distributions.
We demonstrate the effectiveness of abbr through comprehensive experiments on multiple OOD detection benchmarks.
arXiv Detail & Related papers (2023-11-28T02:00:47Z) - ImageNet-OOD: Deciphering Modern Out-of-Distribution Detection Algorithms [27.67837353597245]
Out-of-distribution (OOD) detection is notoriously ill-defined.
Recent works argue for a focus on failure detection.
Complex OOD detectors that were previously considered state-of-the-art now perform similarly to, or even worse than the simple maximum softmax probability baseline.
arXiv Detail & Related papers (2023-10-03T02:37:57Z) - LINe: Out-of-Distribution Detection by Leveraging Important Neurons [15.797257361788812]
We introduce a new aspect for analyzing the difference in model outputs between in-distribution data and OOD data.
We propose a novel method, Leveraging Important Neurons (LINe), for post-hoc Out of distribution detection.
arXiv Detail & Related papers (2023-03-24T13:49:05Z) - Out-of-distribution Detection with Implicit Outlier Transformation [72.73711947366377]
Outlier exposure (OE) is powerful in out-of-distribution (OOD) detection.
We propose a novel OE-based approach that makes the model perform well for unseen OOD situations.
arXiv Detail & Related papers (2023-03-09T04:36:38Z) - Unsupervised Evaluation of Out-of-distribution Detection: A Data-centric
Perspective [55.45202687256175]
Out-of-distribution (OOD) detection methods assume that they have test ground truths, i.e., whether individual test samples are in-distribution (IND) or OOD.
In this paper, we are the first to introduce the unsupervised evaluation problem in OOD detection.
We propose three methods to compute Gscore as an unsupervised indicator of OOD detection performance.
arXiv Detail & Related papers (2023-02-16T13:34:35Z) - Out of Distribution Detection via Neural Network Anchoring [38.36467447555689]
We exploit heteroscedastic temperature scaling as a calibration strategy for out of distribution (OOD) detection.
We propose a new training strategy called anchoring that can estimate appropriate temperature values for each sample.
In contrast to some of the best-performing OOD detection approaches, our method does not require exposure to additional outlier datasets.
arXiv Detail & Related papers (2022-07-08T21:01:09Z) - Meta Learning Low Rank Covariance Factors for Energy-Based Deterministic
Uncertainty [58.144520501201995]
Bi-Lipschitz regularization of neural network layers preserve relative distances between data instances in the feature spaces of each layer.
With the use of an attentive set encoder, we propose to meta learn either diagonal or diagonal plus low-rank factors to efficiently construct task specific covariance matrices.
We also propose an inference procedure which utilizes scaled energy to achieve a final predictive distribution.
arXiv Detail & Related papers (2021-10-12T22:04:19Z) - Types of Out-of-Distribution Texts and How to Detect Them [4.854346360117765]
We categorize OOD examples by whether they exhibit a background shift or a semantic shift.
We find that the two major approaches to OOD detection, model calibration and density estimation, have distinct behavior on these types of OOD data.
Our results call for an explicit definition of OOD examples when evaluating different detection methods.
arXiv Detail & Related papers (2021-09-14T17:12:38Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.