DialogVCS: Robust Natural Language Understanding in Dialogue System Upgrade

• URL: http://arxiv.org/abs/2305.14751v1
• Date: Wed, 24 May 2023 05:53:38 GMT
• Title: DialogVCS: Robust Natural Language Understanding in Dialogue System Upgrade
• Authors: Zefan Cai, Xin Zheng, Tianyu Liu, Xu Wang, Haoran Meng, Jiaqi Han, Gang Yuan, Binghuai Lin, Baobao Chang and Yunbo Cao
• Abstract summary: In constant updates of product dialogue systems, new data from the real users would be merged into the existent data. New intents would emerge and might have semantic entanglement with the existing intents. We setup a new benchmark consisting of 4 Dialogue Version Control dataSets (VCS) We formulate the intent detection with imperfect data in the system update as a multi-label classification task with positive but unlabeled intents.
• Score: 36.433020605744986
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In the constant updates of the product dialogue systems, we need to retrain the natural language understanding (NLU) model as new data from the real users would be merged into the existent data accumulated in the last updates. Within the newly added data, new intents would emerge and might have semantic entanglement with the existing intents, e.g. new intents that are semantically too specific or generic are actually subset or superset of some existing intents in the semantic space, thus impairing the robustness of the NLU model. As the first attempt to solve this problem, we setup a new benchmark consisting of 4 Dialogue Version Control dataSets (DialogVCS). We formulate the intent detection with imperfect data in the system update as a multi-label classification task with positive but unlabeled intents, which asks the models to recognize all the proper intents, including the ones with semantic entanglement, in the inference. We also propose comprehensive baseline models and conduct in-depth analyses for the benchmark, showing that the semantically entangled intents can be effectively recognized with an automatic workflow.

Related papers

• IntentGPT: Few-shot Intent Discovery with Large Language Models [9.245106106117317]
  We develop a model capable of identifying new intents as they emerge. IntentGPT is a training-free method that effectively prompts Large Language Models (LLMs) to discover new intents with minimal labeled data. Our experiments show that IntentGPT outperforms previous methods that require extensive domain-specific data and fine-tuning.
  arXiv  Detail & Related papers  (2024-11-16T02:16:59Z)
• Can Your Model Tell a Negation from an Implicature? Unravelling Challenges With Intent Encoders [24.42199777529863]
  Large Language Models (LLMs) enable embeddings allowing one to adjust semantics over the embedding space using prompts. Traditional evaluation benchmarks rely solely on task metrics that don't particularly measure gaps related to semantic understanding. We propose an intent semantic toolkit that gives a more holistic view of intent embedding models.
  arXiv  Detail & Related papers  (2024-03-07T08:32:17Z)
• Continuously Learning New Words in Automatic Speech Recognition [56.972851337263755]
  We propose an self-supervised continual learning approach to recognize new words. We use a memory-enhanced Automatic Speech Recognition model from previous work. We show that with this approach, we obtain increasing performance on the new words when they occur more frequently.
  arXiv  Detail & Related papers  (2024-01-09T10:39:17Z)
• Learning From Free-Text Human Feedback -- Collect New Datasets Or Extend Existing Ones? [57.16050211534735]
  We investigate the types and frequency of free-text human feedback in commonly used dialog datasets. Our findings provide new insights into the composition of the datasets examined, including error types, user response types, and the relations between them.
  arXiv  Detail & Related papers  (2023-10-24T12:01:11Z)
• SPACE-2: Tree-Structured Semi-Supervised Contrastive Pre-training for Task-Oriented Dialog Understanding [68.94808536012371]
  We propose a tree-structured pre-trained conversation model, which learns dialog representations from limited labeled dialogs and large-scale unlabeled dialog corpora. Our method can achieve new state-of-the-art results on the DialoGLUE benchmark consisting of seven datasets and four popular dialog understanding tasks.
  arXiv  Detail & Related papers  (2022-09-14T13:42:50Z)
• Redwood: Using Collision Detection to Grow a Large-Scale Intent Classification Dataset [4.224157527132053]
  In intent classification systems, problems can arise if training data for a new skill's intent overlaps semantically with an already-existing intent. This paper introduces the task of intent collision detection between multiple datasets for the purposes of growing a system's skillset. To highlight the need for intent collision detection, we show that model performance suffers if new data is added in such a way that does not arbitrate intents.
  arXiv  Detail & Related papers  (2022-04-12T02:28:23Z)
• Synthesizing the Unseen for Zero-shot Object Detection [72.38031440014463]
  We propose to synthesize visual features for unseen classes, so that the model learns both seen and unseen objects in the visual domain. We use a novel generative model that uses class-semantics to not only generate the features but also to discriminatively separate them.
  arXiv  Detail & Related papers  (2020-10-19T12:36:11Z)
• Intent Mining from past conversations for conversational agent [1.9754522186574608]
  Bots are increasingly being deployed to provide round-the-clock support and to increase customer engagement. Many of the commercial bot building frameworks follow a standard approach that requires one to build and train an intent model to recognize a user input. We have introduced a novel density-based clustering algorithm ITERDB-LabelSCAN for unbalanced data clustering.
  arXiv  Detail & Related papers  (2020-05-22T05:29:13Z)
• Automatic Discovery of Novel Intents & Domains from Text Utterances [18.39942131996558]
  We propose a novel framework, ADVIN, to automatically discover novel domains and intents from large volumes of unlabeled data. ADVIN significantly outperforms baselines on three benchmark datasets, and real user utterances from a commercial voice-powered agent.
  arXiv  Detail & Related papers  (2020-05-22T00:47:10Z)
• AGIF: An Adaptive Graph-Interactive Framework for Joint Multiple Intent Detection and Slot Filling [69.59096090788125]
  In this paper, we propose an Adaptive Graph-Interactive Framework (AGIF) for joint multiple intent detection and slot filling. We introduce an intent-slot graph interaction layer to model the strong correlation between the slot and intents. Such an interaction layer is applied to each token adaptively, which has the advantage to automatically extract the relevant intents information.
  arXiv  Detail & Related papers  (2020-04-21T15:07:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.