Improved Out-of-Scope Intent Classification with Dual Encoding and Threshold-based Re-Classification

• URL: http://arxiv.org/abs/2405.19967v2
• Date: Fri, 31 May 2024 08:54:24 GMT
• Title: Improved Out-of-Scope Intent Classification with Dual Encoding and Threshold-based Re-Classification
• Authors: Hossam M. Zawbaa, Wael Rashwan, Sourav Dutta, Haytham Assem,
• Abstract summary: Current methodologies face difficulties with the unpredictable distribution of outliers. We present the Dual for Threshold-Based Re-Classification (DETER) to address these challenges. Our model outperforms previous benchmarks, increasing up to 13% and 5% in F1 score for known and unknown intents.
• Score: 6.975902383951604
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Detecting out-of-scope user utterances is essential for task-oriented dialogues and intent classification. Current methodologies face difficulties with the unpredictable distribution of outliers and often rely on assumptions about data distributions. We present the Dual Encoder for Threshold-Based Re-Classification (DETER) to address these challenges. This end-to-end framework efficiently detects out-of-scope intents without requiring assumptions on data distributions or additional post-processing steps. The core of DETER utilizes dual text encoders, the Universal Sentence Encoder (USE) and the Transformer-based Denoising AutoEncoder (TSDAE), to generate user utterance embeddings, which are classified through a branched neural architecture. Further, DETER generates synthetic outliers using self-supervision and incorporates out-of-scope phrases from open-domain datasets. This approach ensures a comprehensive training set for out-of-scope detection. Additionally, a threshold-based re-classification mechanism refines the model's initial predictions. Evaluations on the CLINC-150, Stackoverflow, and Banking77 datasets demonstrate DETER's efficacy. Our model outperforms previous benchmarks, increasing up to 13% and 5% in F1 score for known and unknown intents on CLINC-150 and Stackoverflow, and 16% for known and 24% % for unknown intents on Banking77. The source code has been released at https://github.com/Hossam-Mohammed-tech/Intent_Classification_OOS.

Related papers

• A new approach for fine-tuning sentence transformers for intent classification and out-of-scope detection tasks [6.013042193107048]
  In virtual assistant systems it is important to reject or redirect user queries that fall outside the scope of the system. One of the most accurate approaches for out-of-scope (OOS) rejection is to combine it with the task of intent classification on in-scope queries. Our work proposes to regularize the cross-entropy loss with an in-scope embedding reconstruction loss learned using an auto-encoder.
  arXiv  Detail & Related papers  (2024-10-17T15:15:12Z)
• Self-Taught Recognizer: Toward Unsupervised Adaptation for Speech Foundation Models [84.8919069953397]
  Self-TAught Recognizer (STAR) is an unsupervised adaptation framework for speech recognition systems. We show that STAR achieves an average of 13.5% relative reduction in word error rate across 14 target domains. STAR exhibits high data efficiency that only requires less than one-hour unlabeled data.
  arXiv  Detail & Related papers  (2024-05-23T04:27:11Z)
• UNFUSED: UNsupervised Finetuning Using SElf supervised Distillation [53.06337011259031]
  We introduce UnFuSeD, a novel approach to leverage self-supervised learning for audio classification. We use the encoder to generate pseudo-labels for unsupervised fine-tuning before the actual fine-tuning step. UnFuSeD achieves state-of-the-art results on the LAPE Benchmark, significantly outperforming all our baselines.
  arXiv  Detail & Related papers  (2023-03-10T02:43:36Z)
• Boosting the Discriminant Power of Naive Bayes [17.43377106246301]
  We propose a feature augmentation method employing a stack auto-encoder to reduce the noise in the data and boost the discriminant power of naive Bayes. The experimental results show that the proposed method significantly and consistently outperforms the state-of-the-art naive Bayes classifiers.
  arXiv  Detail & Related papers  (2022-09-20T08:02:54Z)
• Rethinking Reconstruction Autoencoder-Based Out-of-Distribution Detection [0.0]
  Reconstruction autoencoder-based methods deal with the problem by using input reconstruction error as a metric of novelty vs. normality. We introduce semantic reconstruction, data certainty decomposition and normalized L2 distance to substantially improve original methods. Our method works without any additional data, hard-to-implement structure, time-consuming pipeline, and even harming the classification accuracy of known classes.
  arXiv  Detail & Related papers  (2022-03-04T09:04:55Z)
• End-to-end contextual asr based on posterior distribution adaptation for hybrid ctc/attention system [61.148549738631814]
  End-to-end (E2E) speech recognition architectures assemble all components of traditional speech recognition system into a single model. Although it simplifies ASR system, it introduces contextual ASR drawback: the E2E model has worse performance on utterances containing infrequent proper nouns. We propose to add a contextual bias attention (CBA) module to attention based encoder decoder (AED) model to improve its ability of recognizing the contextual phrases.
  arXiv  Detail & Related papers  (2022-02-18T03:26:02Z)
• Open-Set Recognition: A Good Closed-Set Classifier is All You Need [146.6814176602689]
  We show that the ability of a classifier to make the 'none-of-above' decision is highly correlated with its accuracy on the closed-set classes. We use this correlation to boost the performance of the cross-entropy OSR 'baseline' by improving its closed-set accuracy. We also construct new benchmarks which better respect the task of detecting semantic novelty.
  arXiv  Detail & Related papers  (2021-10-12T17:58:59Z)
• Out-of-Scope Intent Detection with Self-Supervision and Discriminative Training [20.242645823965145]
  Out-of-scope intent detection is of practical importance in task-oriented dialogue systems. We propose a method to train an out-of-scope intent classifier in a fully end-to-end manner by simulating the test scenario in training. We evaluate our method extensively on four benchmark dialogue datasets and observe significant improvements over state-of-the-art approaches.
  arXiv  Detail & Related papers  (2021-06-16T08:17:18Z)
• Encoding Syntactic Knowledge in Transformer Encoder for Intent Detection and Slot Filling [6.234581622120001]
  We propose a novel Transformer encoder-based architecture with syntactical knowledge encoded for intent detection and slot filling. We encode syntactic knowledge into the Transformer encoder by jointly training it to predict syntactic parse ancestors and part-of-speech of each token via multi-task learning.
  arXiv  Detail & Related papers  (2020-12-21T21:25:11Z)
• Discriminative Nearest Neighbor Few-Shot Intent Detection by Transferring Natural Language Inference [150.07326223077405]
  Few-shot learning is attracting much attention to mitigate data scarcity. We present a discriminative nearest neighbor classification with deep self-attention. We propose to boost the discriminative ability by transferring a natural language inference (NLI) model.
  arXiv  Detail & Related papers  (2020-10-25T00:39:32Z)
• Unsupervised Anomaly Detection with Adversarial Mirrored AutoEncoders [51.691585766702744]
  We propose a variant of Adversarial Autoencoder which uses a mirrored Wasserstein loss in the discriminator to enforce better semantic-level reconstruction. We put forward an alternative measure of anomaly score to replace the reconstruction-based metric. Our method outperforms the current state-of-the-art methods for anomaly detection on several OOD detection benchmarks.
  arXiv  Detail & Related papers  (2020-03-24T08:26:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.