Benchmarking Intent Detection for Task-Oriented Dialog Systems
- URL: http://arxiv.org/abs/2012.03929v1
- Date: Mon, 7 Dec 2020 18:58:57 GMT
- Title: Benchmarking Intent Detection for Task-Oriented Dialog Systems
- Authors: Haode Qi, Lin Pan, Atin Sood, Abhishek Shah, Ladislav Kunc, Saloni
Potdar
- Abstract summary: Intent detection is a key component of modern goal-oriented dialog systems that accomplish a user task by predicting the intent of users' text input.
There are three primary challenges in designing robust and accurate intent detection models.
Our results show that Watson Assistant's intent detection model outperforms other commercial solutions.
- Score: 6.54201796167054
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Intent detection is a key component of modern goal-oriented dialog systems
that accomplish a user task by predicting the intent of users' text input.
There are three primary challenges in designing robust and accurate intent
detection models. First, typical intent detection models require a large amount
of labeled data to achieve high accuracy. Unfortunately, in practical scenarios
it is more common to find small, unbalanced, and noisy datasets. Secondly, even
with large training data, the intent detection models can see a different
distribution of test data when being deployed in the real world, leading to
poor accuracy. Finally, a practical intent detection model must be
computationally efficient in both training and single query inference so that
it can be used continuously and re-trained frequently. We benchmark intent
detection methods on a variety of datasets. Our results show that Watson
Assistant's intent detection model outperforms other commercial solutions and
is comparable to large pretrained language models while requiring only a
fraction of computational resources and training data. Watson Assistant
demonstrates a higher degree of robustness when the training and test
distributions differ.
Related papers
- Learning 3D Perception from Others' Predictions [64.09115694891679]
We investigate a new scenario to construct 3D object detectors: learning from the predictions of a nearby unit that is equipped with an accurate detector.
For example, when a self-driving car enters a new area, it may learn from other traffic participants whose detectors have been optimized for that area.
arXiv Detail & Related papers (2024-10-03T16:31:28Z) - Generalized Few-Shot 3D Object Detection of LiDAR Point Cloud for
Autonomous Driving [91.39625612027386]
We propose a novel task, called generalized few-shot 3D object detection, where we have a large amount of training data for common (base) objects, but only a few data for rare (novel) classes.
Specifically, we analyze in-depth differences between images and point clouds, and then present a practical principle for the few-shot setting in the 3D LiDAR dataset.
To solve this task, we propose an incremental fine-tuning method to extend existing 3D detection models to recognize both common and rare objects.
arXiv Detail & Related papers (2023-02-08T07:11:36Z) - Reinforcement Guided Multi-Task Learning Framework for Low-Resource
Stereotype Detection [3.7223111129285096]
"Stereotype Detection" datasets mainly adopt a diagnostic approach toward large Pre-trained Language Models.
Annotating a reliable dataset requires a precise understanding of the subtle nuances of how stereotypes manifest in text.
We present a multi-task model that leverages the abundance of data-rich neighboring tasks to improve the empirical performance on "Stereotype Detection"
arXiv Detail & Related papers (2022-03-27T17:16:11Z) - Out-of-Scope Intent Detection with Self-Supervision and Discriminative
Training [20.242645823965145]
Out-of-scope intent detection is of practical importance in task-oriented dialogue systems.
We propose a method to train an out-of-scope intent classifier in a fully end-to-end manner by simulating the test scenario in training.
We evaluate our method extensively on four benchmark dialogue datasets and observe significant improvements over state-of-the-art approaches.
arXiv Detail & Related papers (2021-06-16T08:17:18Z) - ALT-MAS: A Data-Efficient Framework for Active Testing of Machine
Learning Algorithms [58.684954492439424]
We propose a novel framework to efficiently test a machine learning model using only a small amount of labeled test data.
The idea is to estimate the metrics of interest for a model-under-test using Bayesian neural network (BNN)
arXiv Detail & Related papers (2021-04-11T12:14:04Z) - Generalized Zero-shot Intent Detection via Commonsense Knowledge [5.398580049917152]
We propose RIDE: an intent detection model that leverages commonsense knowledge in an unsupervised fashion to overcome the issue of training data scarcity.
RIDE computes robust and generalizable relationship meta-features that capture deep semantic relationships between utterances and intent labels.
Our extensive experimental analysis on three widely-used intent detection benchmarks shows that relationship meta-features significantly increase the accuracy of detecting both seen and unseen intents.
arXiv Detail & Related papers (2021-02-04T23:36:41Z) - Discriminative Nearest Neighbor Few-Shot Intent Detection by
Transferring Natural Language Inference [150.07326223077405]
Few-shot learning is attracting much attention to mitigate data scarcity.
We present a discriminative nearest neighbor classification with deep self-attention.
We propose to boost the discriminative ability by transferring a natural language inference (NLI) model.
arXiv Detail & Related papers (2020-10-25T00:39:32Z) - Intent Detection with WikiHow [28.28719498563396]
Our models are able to predict a broad range of intended goals from many actions because they are trained on wikiHow.
Our models achieve state-of-the-art results on the Snips dataset, theGuided Dialogue dataset, and all 3 languages of the Facebook multilingual dialog datasets.
arXiv Detail & Related papers (2020-09-12T12:53:47Z) - Efficient Intent Detection with Dual Sentence Encoders [53.16532285820849]
We introduce intent detection methods backed by pretrained dual sentence encoders such as USE and ConveRT.
We demonstrate the usefulness and wide applicability of the proposed intent detectors, showing that they outperform intent detectors based on fine-tuning the full BERT-Large model.
We release our code, as well as a new challenging single-domain intent detection dataset.
arXiv Detail & Related papers (2020-03-10T15:33:54Z) - Stance Detection Benchmark: How Robust Is Your Stance Detection? [65.91772010586605]
Stance Detection (StD) aims to detect an author's stance towards a certain topic or claim.
We introduce a StD benchmark that learns from ten StD datasets of various domains in a multi-dataset learning setting.
Within this benchmark setup, we are able to present new state-of-the-art results on five of the datasets.
arXiv Detail & Related papers (2020-01-06T13:37:51Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.