A Hybrid Task-Oriented Dialog System with Domain and Task Adaptive Pretraining

• URL: http://arxiv.org/abs/2102.04506v1
• Date: Mon, 8 Feb 2021 20:02:30 GMT
• Title: A Hybrid Task-Oriented Dialog System with Domain and Task Adaptive Pretraining
• Authors: Boliang Zhang, Ying Lyu, Ning Ding, Tianhao Shen, Zhaoyang Jia, Kun Han, Kevin Knight
• Abstract summary: This paper describes our submission for the End-to-end Multi-domain Task Completion Dialog shared task at the 9th Dialog System Technology Challenge (DSTC-9) Participants in the shared task build an end-to-end task completion dialog system which is evaluated by human evaluation and a user simulator based automatic evaluation.
• Score: 25.674966922466467
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper describes our submission for the End-to-end Multi-domain Task Completion Dialog shared task at the 9th Dialog System Technology Challenge (DSTC-9). Participants in the shared task build an end-to-end task completion dialog system which is evaluated by human evaluation and a user simulator based automatic evaluation. Different from traditional pipelined approaches where modules are optimized individually and suffer from cascading failure, we propose an end-to-end dialog system that 1) uses Generative Pretraining 2 (GPT-2) as the backbone to jointly solve Natural Language Understanding, Dialog State Tracking, and Natural Language Generation tasks, 2) adopts Domain and Task Adaptive Pretraining to tailor GPT-2 to the dialog domain before finetuning, 3) utilizes heuristic pre/post-processing rules that greatly simplify the prediction tasks and improve generalizability, and 4) equips a fault tolerance module to correct errors and inappropriate responses. Our proposed method significantly outperforms baselines and ties for first place in the official evaluation. We make our source code publicly available.

Related papers

• LEEETs-Dial: Linguistic Entrainment in End-to-End Task-oriented Dialogue systems [0.0]
  We introduce methods for achieving dialogue entrainment in a GPT-2-based end-to-end task-oriented dialogue system. We experiment with training instance weighting, entrainment-specific loss, and additional conditioning to generate responses that align with the user.
  arXiv  Detail & Related papers  (2023-11-15T21:35:25Z)
• Zero-Shot Generalizable End-to-End Task-Oriented Dialog System using Context Summarization and Domain Schema [2.7178968279054936]
  State-of-the-art approaches in task-oriented dialog systems formulate the problem as a conditional sequence generation task. This requires labeled training data for each new domain or task. We introduce a novel Zero-Shot generalizable end-to-end Task-oriented Dialog system, ZS-ToD.
  arXiv  Detail & Related papers  (2023-03-28T18:56:31Z)
• SPACE-2: Tree-Structured Semi-Supervised Contrastive Pre-training for Task-Oriented Dialog Understanding [68.94808536012371]
  We propose a tree-structured pre-trained conversation model, which learns dialog representations from limited labeled dialogs and large-scale unlabeled dialog corpora. Our method can achieve new state-of-the-art results on the DialoGLUE benchmark consisting of seven datasets and four popular dialog understanding tasks.
  arXiv  Detail & Related papers  (2022-09-14T13:42:50Z)
• OPAL: Ontology-Aware Pretrained Language Model for End-to-End Task-Oriented Dialogue [40.62090743056549]
  This paper presents an ontology-aware pretrained language model (OPAL) for end-to-end task-oriented dialogue (TOD) Unlike chit-chat dialogue models, task-oriented dialogue models fulfill at least two task-specific modules: dialogue state tracker (DST) and response generator (RG)
  arXiv  Detail & Related papers  (2022-09-10T04:38:27Z)
• Manual-Guided Dialogue for Flexible Conversational Agents [84.46598430403886]
  How to build and use dialogue data efficiently, and how to deploy models in different domains at scale can be critical issues in building a task-oriented dialogue system. We propose a novel manual-guided dialogue scheme, where the agent learns the tasks from both dialogue and manuals. Our proposed scheme reduces the dependence of dialogue models on fine-grained domain ontology, and makes them more flexible to adapt to various domains.
  arXiv  Detail & Related papers  (2022-08-16T08:21:12Z)
• GODEL: Large-Scale Pre-Training for Goal-Directed Dialog [119.1397031992088]
  We introduce GODEL, a large pre-trained language model for dialog. We show that GODEL outperforms state-of-the-art pre-trained dialog models in few-shot fine-tuning setups. A novel feature of our evaluation methodology is the introduction of a notion of utility that assesses the usefulness of responses.
  arXiv  Detail & Related papers  (2022-06-22T18:19:32Z)
• GALAXY: A Generative Pre-trained Model for Task-Oriented Dialog with Semi-Supervised Learning and Explicit Policy Injection [36.77204909711832]
  We propose a novel pre-trained dialog model that explicitly learns dialog policy from limited labeled dialogs and large-scale unlabeled dialog corpora. Specifically, we introduce a dialog act prediction task for policy optimization during pre-training and employ a consistency regularization term to refine the learned representation. Empirical results show that GALAXY substantially improves the performance of task-oriented dialog systems.
  arXiv  Detail & Related papers  (2021-11-29T15:24:36Z)
• A Unified Pre-training Framework for Conversational AI [25.514505462661763]
  PLATO-2 is trained via two-stage curriculum learning to fit the simplified one-to-one mapping relationship. PLATO-2 has obtained the 1st place in all three tasks, verifying its effectiveness as a unified framework for various dialogue systems.
  arXiv  Detail & Related papers  (2021-05-06T07:27:11Z)
• A Simple Language Model for Task-Oriented Dialogue [61.84084939472287]
  SimpleTOD is a simple approach to task-oriented dialogue that uses a single, causal language model trained on all sub-tasks recast as a single sequence prediction problem. This allows SimpleTOD to fully leverage transfer learning from pre-trained, open domain, causal language models such as GPT-2.
  arXiv  Detail & Related papers  (2020-05-02T11:09:27Z)
• A Tailored Pre-Training Model for Task-Oriented Dialog Generation [60.05269529832447]
  We propose a Pre-trained Role Alternating Language model (PRAL) for task-oriented conversational systems. We introduce a task-oriented dialog pretraining dataset by cleaning 13 existing data sets. The results show that PRAL performs better or on par with state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-04-24T09:25:45Z)
• TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue [113.45485470103762]
  In this work, we unify nine human-human and multi-turn task-oriented dialogue datasets for language modeling. To better model dialogue behavior during pre-training, we incorporate user and system tokens into the masked language modeling.
  arXiv  Detail & Related papers  (2020-04-15T04:09:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.