Enhancing Out-of-Distribution Detection in Natural Language Understanding via Implicit Layer Ensemble

• URL: http://arxiv.org/abs/2210.11034v1
• Date: Thu, 20 Oct 2022 06:05:58 GMT
• Title: Enhancing Out-of-Distribution Detection in Natural Language Understanding via Implicit Layer Ensemble
• Authors: Hyunsoo Cho, Choonghyun Park, Jaewook Kang, Kang Min Yoo, Taeuk Kim, Sang-goo Lee
• Abstract summary: Out-of-distribution (OOD) detection aims to discern outliers from the intended data distribution. We propose a novel framework based on contrastive learning that encourages intermediate features to learn layer-specialized representations. Our approach is significantly more effective than other works.
• Score: 22.643719584452455
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Out-of-distribution (OOD) detection aims to discern outliers from the intended data distribution, which is crucial to maintaining high reliability and a good user experience. Most recent studies in OOD detection utilize the information from a single representation that resides in the penultimate layer to determine whether the input is anomalous or not. Although such a method is straightforward, the potential of diverse information in the intermediate layers is overlooked. In this paper, we propose a novel framework based on contrastive learning that encourages intermediate features to learn layer-specialized representations and assembles them implicitly into a single representation to absorb rich information in the pre-trained language model. Extensive experiments in various intent classification and OOD datasets demonstrate that our approach is significantly more effective than other works.

Related papers

• FlowCon: Out-of-Distribution Detection using Flow-Based Contrastive Learning [0.0]
  We introduce textitFlowCon, a new density-based OOD detection technique. Our main innovation lies in efficiently combining the properties of normalizing flow with supervised contrastive learning. Empirical evaluation shows the enhanced performance of our method across common vision datasets.
  arXiv  Detail & Related papers  (2024-07-03T20:33:56Z)
• WeiPer: OOD Detection using Weight Perturbations of Class Projections [11.130659240045544]
  We introduce perturbations of the class projections in the final fully connected layer which creates a richer representation of the input. We achieve state-of-the-art OOD detection results across multiple benchmarks of the OpenOOD framework.
  arXiv  Detail & Related papers  (2024-05-27T13:38:28Z)
• Out-of-Distribution Detection Using Peer-Class Generated by Large Language Model [0.0]
  Out-of-distribution (OOD) detection is a critical task to ensure the reliability and security of machine learning models. In this paper, a novel method called ODPC is proposed, in which specific prompts to generate OOD peer classes of ID semantics are designed by a large language model. Experiments on five benchmark datasets show that the method we propose can yield state-of-the-art results.
  arXiv  Detail & Related papers  (2024-03-20T06:04:05Z)
• EAT: Towards Long-Tailed Out-of-Distribution Detection [55.380390767978554]
  This paper addresses the challenging task of long-tailed OOD detection. The main difficulty lies in distinguishing OOD data from samples belonging to the tail classes. We propose two simple ideas: (1) Expanding the in-distribution class space by introducing multiple abstention classes, and (2) Augmenting the context-limited tail classes by overlaying images onto the context-rich OOD data.
  arXiv  Detail & Related papers  (2023-12-14T13:47:13Z)
• ExCeL : Combined Extreme and Collective Logit Information for Enhancing Out-of-Distribution Detection [9.689089164964484]
  ExCeL combines extreme and collective information within the output layer for enhanced accuracy in OOD detection. We show that ExCeL consistently is among the five top-performing methods out of twenty-one existing post-hoc baselines.
  arXiv  Detail & Related papers  (2023-11-23T14:16:03Z)
• Diversified Outlier Exposure for Out-of-Distribution Detection via Informative Extrapolation [110.34982764201689]
  Out-of-distribution (OOD) detection is important for deploying reliable machine learning models on real-world applications. Recent advances in outlier exposure have shown promising results on OOD detection via fine-tuning model with informatively sampled auxiliary outliers. We propose a novel framework, namely, Diversified Outlier Exposure (DivOE), for effective OOD detection via informative extrapolation based on the given auxiliary outliers.
  arXiv  Detail & Related papers  (2023-10-21T07:16:09Z)
• Exploring Large Language Models for Multi-Modal Out-of-Distribution Detection [67.68030805755679]
  Large language models (LLMs) encode a wealth of world knowledge and can be prompted to generate descriptive features for each class. In this paper, we propose to apply world knowledge to enhance OOD detection performance through selective generation from LLMs.
  arXiv  Detail & Related papers  (2023-10-12T04:14:28Z)
• Class Relevance Learning For Out-of-distribution Detection [16.029229052068]
  This paper presents an innovative class relevance learning method tailored for OOD detection. Our method establishes a comprehensive class relevance learning framework, strategically harnessing interclass relationships within the OOD pipeline.
  arXiv  Detail & Related papers  (2023-09-21T08:38:21Z)
• Unleashing Mask: Explore the Intrinsic Out-of-Distribution Detection Capability [70.72426887518517]
  Out-of-distribution (OOD) detection is an indispensable aspect of secure AI when deploying machine learning models in real-world applications. We propose a novel method, Unleashing Mask, which aims to restore the OOD discriminative capabilities of the well-trained model with ID data. Our method utilizes a mask to figure out the memorized atypical samples, and then finetune the model or prune it with the introduced mask to forget them.
  arXiv  Detail & Related papers  (2023-06-06T14:23:34Z)
• Breaking Down Out-of-Distribution Detection: Many Methods Based on OOD Training Data Estimate a Combination of the Same Core Quantities [104.02531442035483]
  The goal of this paper is to recognize common objectives as well as to identify the implicit scoring functions of different OOD detection methods. We show that binary discrimination between in- and (different) out-distributions is equivalent to several distinct formulations of the OOD detection problem. We also show that the confidence loss which is used by Outlier Exposure has an implicit scoring function which differs in a non-trivial fashion from the theoretically optimal scoring function.
  arXiv  Detail & Related papers  (2022-06-20T16:32:49Z)
• Triggering Failures: Out-Of-Distribution detection by learning from local adversarial attacks in Semantic Segmentation [76.2621758731288]
  We tackle the detection of out-of-distribution (OOD) objects in semantic segmentation. Our main contribution is a new OOD detection architecture called ObsNet associated with a dedicated training scheme based on Local Adversarial Attacks (LAA) We show it obtains top performances both in speed and accuracy when compared to ten recent methods of the literature on three different datasets.
  arXiv  Detail & Related papers  (2021-08-03T17:09:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.