BED: Bi-Encoder-Based Detectors for Out-of-Distribution Detection

• URL: http://arxiv.org/abs/2306.08852v2
• Date: Wed, 13 Mar 2024 08:49:54 GMT
• Title: BED: Bi-Encoder-Based Detectors for Out-of-Distribution Detection
• Authors: Louis Owen, Biddwan Ahmed, Abhay Kumar
• Abstract summary: This paper introduces a novel method leveraging bi-encoder-based detectors. A comprehensive study comparing different out-of-distribution (OOD) detection methods in NLP is conducted. The proposed bi-encoder-based detectors outperform other methods, both those that require OOD labels in training and those that do not. The simplicity of the training process and the superior detection performance make them applicable to real-world scenarios.
• Score: 0.43891501568660135
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper introduces a novel method leveraging bi-encoder-based detectors along with a comprehensive study comparing different out-of-distribution (OOD) detection methods in NLP using different feature extractors. The feature extraction stage employs popular methods such as Universal Sentence Encoder (USE), BERT, MPNET, and GLOVE to extract informative representations from textual data. The evaluation is conducted on several datasets, including CLINC150, ROSTD-Coarse, SNIPS, and YELLOW. Performance is assessed using metrics such as F1-Score, MCC, FPR@90, FPR@95, AUPR, an AUROC. The experimental results demonstrate that the proposed bi-encoder-based detectors outperform other methods, both those that require OOD labels in training and those that do not, across all datasets, showing great potential for OOD detection in NLP. The simplicity of the training process and the superior detection performance make them applicable to real-world scenarios. The presented methods and benchmarking metrics serve as a valuable resource for future research in OOD detection, enabling further advancements in this field. The code and implementation details can be found on our GitHub repository: https://github.com/yellowmessenger/ood-detection.

Related papers

• Collaborative Feature-Logits Contrastive Learning for Open-Set Semi-Supervised Object Detection [75.02249869573994]
  In open-set scenarios, the unlabeled dataset contains both in-distribution (ID) classes and out-of-distribution (OOD) classes. Applying semi-supervised detectors in such settings can lead to misclassifying OOD class as ID classes. We propose a simple yet effective method, termed Collaborative Feature-Logits Detector (CFL-Detector)
  arXiv  Detail & Related papers  (2024-11-20T02:57:35Z)
• Improving Out-of-Distribution Detection by Combining Existing Post-hoc Methods [1.747623282473278]
  Post-hoc deep Out-of-Distribution (OOD) detection has expanded rapidly. Current best practice is to test all the methods on the datasets at hand. This paper shifts focus from developing new methods to effectively combining existing ones to enhance OOD detection.
  arXiv  Detail & Related papers  (2024-07-09T15:46:39Z)
• Envisioning Outlier Exposure by Large Language Models for Out-of-Distribution Detection [71.93411099797308]
  Out-of-distribution (OOD) samples are crucial when deploying machine learning models in open-world scenarios. We propose to tackle this constraint by leveraging the expert knowledge and reasoning capability of large language models (LLM) to potential Outlier Exposure, termed EOE. EOE can be generalized to different tasks, including far, near, and fine-language OOD detection. EOE achieves state-of-the-art performance across different OOD tasks and can be effectively scaled to the ImageNet-1K dataset.
  arXiv  Detail & Related papers  (2024-06-02T17:09:48Z)
• Nearest Neighbor Guidance for Out-of-Distribution Detection [18.851275688720108]
  We propose Nearest Neighbor Guidance (NNGuide) for detecting out-of-distribution (OOD) samples. NNGuide reduces the overconfidence of OOD samples while preserving the fine-grained capability of the classifier-based score. Our results demonstrate that NNGuide provides a significant performance improvement on the base detection scores.
  arXiv  Detail & Related papers  (2023-09-26T12:40:35Z)
• Beyond AUROC & co. for evaluating out-of-distribution detection performance [50.88341818412508]
  Given their relevance for safe(r) AI, it is important to examine whether the basis for comparing OOD detection methods is consistent with practical needs. We propose a new metric - Area Under the Threshold Curve (AUTC), which explicitly penalizes poor separation between ID and OOD samples.
  arXiv  Detail & Related papers  (2023-06-26T12:51:32Z)
• A Functional Data Perspective and Baseline On Multi-Layer Out-of-Distribution Detection [30.499548939422194]
  Methods that explore the multiple layers either require a special architecture or a supervised objective to do so. This work adopts an original approach based on a functional view of the network that exploits the sample's trajectories through the various layers and their statistical dependencies. We validate our method and empirically demonstrate its effectiveness in OOD detection compared to strong state-of-the-art baselines on computer vision benchmarks.
  arXiv  Detail & Related papers  (2023-06-06T09:14:05Z)
• Unsupervised Evaluation of Out-of-distribution Detection: A Data-centric Perspective [55.45202687256175]
  Out-of-distribution (OOD) detection methods assume that they have test ground truths, i.e., whether individual test samples are in-distribution (IND) or OOD. In this paper, we are the first to introduce the unsupervised evaluation problem in OOD detection. We propose three methods to compute Gscore as an unsupervised indicator of OOD detection performance.
  arXiv  Detail & Related papers  (2023-02-16T13:34:35Z)
• Beyond Mahalanobis-Based Scores for Textual OOD Detection [32.721317681946246]
  We introduce TRUSTED, a new OOD detector for classifiers based on Transformer architectures that meets operational requirements. The efficiency of TRUSTED relies on the fruitful idea that all hidden layers carry relevant information to detect OOD examples. Our experiments involve 51k model configurations, including various checkpoints, seeds, datasets, and demonstrate that TRUSTED achieves state-of-the-art performances.
  arXiv  Detail & Related papers  (2022-11-24T10:51:58Z)
• Prompt-driven efficient Open-set Semi-supervised Learning [52.30303262499391]
  Open-set semi-supervised learning (OSSL) has attracted growing interest, which investigates a more practical scenario where out-of-distribution (OOD) samples are only contained in unlabeled data. We propose a prompt-driven efficient OSSL framework, called OpenPrompt, which can propagate class information from labeled to unlabeled data with only a small number of trainable parameters.
  arXiv  Detail & Related papers  (2022-09-28T16:25:08Z)
• Gradient-based Novelty Detection Boosted by Self-supervised Binary Classification [20.715158729811755]
  Novelty detection aims to automatically identify out-of-distribution (OOD) data, without any prior knowledge of them. We propose a novel, self-supervised approach that does not rely on any pre-defined OOD data. In the evaluation with multiple datasets, the proposed approach consistently outperforms state-of-the-art supervised and unsupervised methods.
  arXiv  Detail & Related papers  (2021-12-18T01:17:15Z)
• Triggering Failures: Out-Of-Distribution detection by learning from local adversarial attacks in Semantic Segmentation [76.2621758731288]
  We tackle the detection of out-of-distribution (OOD) objects in semantic segmentation. Our main contribution is a new OOD detection architecture called ObsNet associated with a dedicated training scheme based on Local Adversarial Attacks (LAA) We show it obtains top performances both in speed and accuracy when compared to ten recent methods of the literature on three different datasets.
  arXiv  Detail & Related papers  (2021-08-03T17:09:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.