Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue System

• URL: http://arxiv.org/abs/2109.14739v1
• Date: Wed, 29 Sep 2021 22:02:18 GMT
• Title: Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue System
• Authors: Yixuan Su, Lei Shu, Elman Mansimov, Arshit Gupta, Deng Cai, Yi-An Lai, Yi Zhang
• Abstract summary: PPTOD is a unified plug-and-play model for task-oriented dialogue. We extensively test our model on three benchmark TOD tasks, including end-to-end dialogue modelling, dialogue state tracking, and intent classification.
• Score: 26.837972034630003
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Pre-trained language models have been recently shown to benefit task-oriented dialogue (TOD) systems. Despite their success, existing methods often formulate this task as a cascaded generation problem which can lead to error accumulation across different sub-tasks and greater data annotation overhead. In this study, we present PPTOD, a unified plug-and-play model for task-oriented dialogue. In addition, we introduce a new dialogue multi-task pre-training strategy that allows the model to learn the primary TOD task completion skills from heterogeneous dialog corpora. We extensively test our model on three benchmark TOD tasks, including end-to-end dialogue modelling, dialogue state tracking, and intent classification. Experimental results show that PPTOD achieves new state of the art on all evaluated tasks in both high-resource and low-resource scenarios. Furthermore, comparisons against previous SOTA methods show that the responses generated by PPTOD are more factually correct and semantically coherent as judged by human annotators.

Related papers

• Multitask Multimodal Prompted Training for Interactive Embodied Task Completion [48.69347134411864]
  Embodied MultiModal Agent (EMMA) is a unified encoder-decoder model that reasons over images and trajectories. By unifying all tasks as text generation, EMMA learns a language of actions which facilitates transfer across tasks.
  arXiv  Detail & Related papers  (2023-11-07T15:27:52Z)
• InstructTODS: Large Language Models for End-to-End Task-Oriented Dialogue Systems [60.53276524369498]
  Large language models (LLMs) have been used for diverse tasks in natural language processing (NLP) We present InstructTODS, a novel framework for zero-shot end-to-end task-oriented dialogue systems. InstructTODS generates a proxy belief state that seamlessly translates user intentions into dynamic queries.
  arXiv  Detail & Related papers  (2023-10-13T06:36:26Z)
• A Task-oriented Dialog Model with Task-progressive and Policy-aware Pre-training [10.766299050844603]
  This paper proposes a task-progressive PCM with two policy-aware pre-training tasks. A global policy consistency task is designed to capture the multi-turn dialog policy sequential relation. An act-based contrastive learning task is designed to capture similarities among samples with the same dialog policy.
  arXiv  Detail & Related papers  (2023-10-01T07:06:02Z)
• OPAL: Ontology-Aware Pretrained Language Model for End-to-End Task-Oriented Dialogue [40.62090743056549]
  This paper presents an ontology-aware pretrained language model (OPAL) for end-to-end task-oriented dialogue (TOD) Unlike chit-chat dialogue models, task-oriented dialogue models fulfill at least two task-specific modules: dialogue state tracker (DST) and response generator (RG)
  arXiv  Detail & Related papers  (2022-09-10T04:38:27Z)
• A Multi-Task BERT Model for Schema-Guided Dialogue State Tracking [78.2700757742992]
  Task-oriented dialogue systems often employ a Dialogue State Tracker (DST) to successfully complete conversations. Recent state-of-the-art DST implementations rely on schemata of diverse services to improve model robustness. We propose a single multi-task BERT-based model that jointly solves the three DST tasks of intent prediction, requested slot prediction and slot filling.
  arXiv  Detail & Related papers  (2022-07-02T13:27:59Z)
• DialogZoo: Large-Scale Dialog-Oriented Task Learning [52.18193690394549]
  We aim to build a unified foundation model which can solve massive diverse dialogue tasks. To achieve this goal, we first collect a large-scale well-labeled dialogue dataset from 73 publicly available datasets.
  arXiv  Detail & Related papers  (2022-05-25T11:17:16Z)
• A Study on Prompt-based Few-Shot Learning Methods for Belief State Tracking in Task-oriented Dialog Systems [10.024834304960846]
  We tackle the Dialogue Belief State Tracking problem of task-oriented conversational systems. Recent approaches to this problem leveraging Transformer-based models have yielded great results. We explore prompt-based few-shot learning for Dialogue Belief State Tracking.
  arXiv  Detail & Related papers  (2022-04-18T05:29:54Z)
• CINS: Comprehensive Instruction for Few-shot Learning in Task-oriented Dialog Systems [56.302581679816775]
  This paper proposes Comprehensive Instruction (CINS) that exploits PLMs with task-specific instructions. We design a schema (definition, constraint, prompt) of instructions and their customized realizations for three important downstream tasks in ToD. Experiments are conducted on these ToD tasks in realistic few-shot learning scenarios with small validation data.
  arXiv  Detail & Related papers  (2021-09-10T03:23:06Z)
• A Tailored Pre-Training Model for Task-Oriented Dialog Generation [60.05269529832447]
  We propose a Pre-trained Role Alternating Language model (PRAL) for task-oriented conversational systems. We introduce a task-oriented dialog pretraining dataset by cleaning 13 existing data sets. The results show that PRAL performs better or on par with state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-04-24T09:25:45Z)
• TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue [113.45485470103762]
  In this work, we unify nine human-human and multi-turn task-oriented dialogue datasets for language modeling. To better model dialogue behavior during pre-training, we incorporate user and system tokens into the masked language modeling.
  arXiv  Detail & Related papers  (2020-04-15T04:09:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.