Unsupervised Layer-wise Score Aggregation for Textual OOD Detection

• URL: http://arxiv.org/abs/2302.09852v3
• Date: Wed, 21 Feb 2024 17:47:37 GMT
• Title: Unsupervised Layer-wise Score Aggregation for Textual OOD Detection
• Authors: Maxime Darrin, Guillaume Staerman, Eduardo Dadalto C\^amara Gomes, Jackie CK Cheung, Pablo Piantanida, Pierre Colombo
• Abstract summary: We observe that OOD detection performance varies greatly depending on the task and layer output. We propose a data-driven, unsupervised method to combine layer-wise anomaly scores. We extend classical OOD benchmarks by including classification tasks with a greater number of classes.
• Score: 35.47177259803885
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Out-of-distribution (OOD) detection is a rapidly growing field due to new robustness and security requirements driven by an increased number of AI-based systems. Existing OOD textual detectors often rely on an anomaly score (e.g., Mahalanobis distance) computed on the embedding output of the last layer of the encoder. In this work, we observe that OOD detection performance varies greatly depending on the task and layer output. More importantly, we show that the usual choice (the last layer) is rarely the best one for OOD detection and that far better results could be achieved if the best layer were picked. To leverage this observation, we propose a data-driven, unsupervised method to combine layer-wise anomaly scores. In addition, we extend classical textual OOD benchmarks by including classification tasks with a greater number of classes (up to 77), which reflects more realistic settings. On this augmented benchmark, we show that the proposed post-aggregation methods achieve robust and consistent results while removing manual feature selection altogether. Their performance achieves near oracle's best layer performance.

Related papers

• The Best of Both Worlds: On the Dilemma of Out-of-distribution Detection [75.65876949930258]
  Out-of-distribution (OOD) detection is essential for model trustworthiness. We show that the superior OOD detection performance of state-of-the-art methods is achieved by secretly sacrificing the OOD generalization ability.
  arXiv  Detail & Related papers  (2024-10-12T07:02:04Z)
• Resultant: Incremental Effectiveness on Likelihood for Unsupervised Out-of-Distribution Detection [63.93728560200819]
  Unsupervised out-of-distribution (U-OOD) detection is to identify data samples with a detector trained solely on unlabeled in-distribution (ID) data. Recent studies have developed various detectors based on DGMs to move beyond likelihood. We apply two techniques for each direction, specifically post-hoc prior and dataset entropy-mutual calibration. Experimental results demonstrate that the Resultant could be a new state-of-the-art U-OOD detector.
  arXiv  Detail & Related papers  (2024-09-05T02:58:13Z)
• Unifying Unsupervised Graph-Level Anomaly Detection and Out-of-Distribution Detection: A Benchmark [73.58840254552656]
  Unsupervised graph-level anomaly detection (GLAD) and unsupervised graph-level out-of-distribution (OOD) detection have received significant attention in recent years. We present a Unified Benchmark for unsupervised Graph-level OOD and anomaly Detection (our method) Our benchmark encompasses 35 datasets spanning four practical anomaly and OOD detection scenarios. We conduct multi-dimensional analyses to explore the effectiveness, generalizability, robustness, and efficiency of existing methods.
  arXiv  Detail & Related papers  (2024-06-21T04:07:43Z)
• Rethinking the Evaluation of Out-of-Distribution Detection: A Sorites Paradox [70.57120710151105]
  Most existing out-of-distribution (OOD) detection benchmarks classify samples with novel labels as the OOD data. Some marginal OOD samples actually have close semantic contents to the in-distribution (ID) sample, which makes determining the OOD sample a Sorites Paradox. We construct a benchmark named Incremental Shift OOD (IS-OOD) to address the issue.
  arXiv  Detail & Related papers  (2024-06-14T09:27:56Z)
• WeiPer: OOD Detection using Weight Perturbations of Class Projections [11.130659240045544]
  We introduce perturbations of the class projections in the final fully connected layer which creates a richer representation of the input. We achieve state-of-the-art OOD detection results across multiple benchmarks of the OpenOOD framework.
  arXiv  Detail & Related papers  (2024-05-27T13:38:28Z)
• EAT: Towards Long-Tailed Out-of-Distribution Detection [55.380390767978554]
  This paper addresses the challenging task of long-tailed OOD detection. The main difficulty lies in distinguishing OOD data from samples belonging to the tail classes. We propose two simple ideas: (1) Expanding the in-distribution class space by introducing multiple abstention classes, and (2) Augmenting the context-limited tail classes by overlaying images onto the context-rich OOD data.
  arXiv  Detail & Related papers  (2023-12-14T13:47:13Z)
• Nearest Neighbor Guidance for Out-of-Distribution Detection [18.851275688720108]
  We propose Nearest Neighbor Guidance (NNGuide) for detecting out-of-distribution (OOD) samples. NNGuide reduces the overconfidence of OOD samples while preserving the fine-grained capability of the classifier-based score. Our results demonstrate that NNGuide provides a significant performance improvement on the base detection scores.
  arXiv  Detail & Related papers  (2023-09-26T12:40:35Z)
• Holistic Sentence Embeddings for Better Out-of-Distribution Detection [12.640837452980332]
  We propose a simple embedding approach named Avg-Avg, which averages all token representations from each intermediate layer as the sentence embedding. Our analysis demonstrates that it indeed helps preserve general linguistic knowledge in fine-tuned PLMs and substantially benefits detecting background shifts.
  arXiv  Detail & Related papers  (2022-10-14T03:22:58Z)
• Semantically Coherent Out-of-Distribution Detection [26.224146828317277]
  Current out-of-distribution (OOD) detection benchmarks are commonly built by defining one dataset as in-distribution (ID) and all others as OOD. We re-design the benchmarks and propose the semantically coherent out-of-distribution detection (SC-OOD) Our approach achieves state-of-the-art performance on SC-OOD benchmarks.
  arXiv  Detail & Related papers  (2021-08-26T17:53:32Z)
• MOOD: Multi-level Out-of-distribution Detection [13.207044902083057]
  Out-of-distribution (OOD) detection is essential to prevent anomalous inputs from causing a model to fail during deployment. We propose a novel framework, multi-level out-of-distribution detection MOOD, which exploits intermediate classifier outputs for dynamic and efficient OOD inference. MOOD achieves up to 71.05% computational reduction in inference, while maintaining competitive OOD detection performance.
  arXiv  Detail & Related papers  (2021-04-30T02:18:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.