OmniDialog: An Omnipotent Pre-training Model for Task-Oriented Dialogue System

• URL: http://arxiv.org/abs/2312.16864v1
• Date: Thu, 28 Dec 2023 07:20:49 GMT
• Title: OmniDialog: An Omnipotent Pre-training Model for Task-Oriented Dialogue System
• Authors: Mingtao Yang, See-Kiong Ng, Jinlan Fu
• Abstract summary: We propose an Omnipotent Dialogue pre-training model ( OmniDialog) It unifies three dialogue tasks into a monolithic framework by multi-task learning, fostering inter-task communication. We evaluate its performance across four tasks: dialogue summarization, end-to-end dialogue modeling, dialogue state tracking, and intent classification.
• Score: 43.92593448255296
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Pre-trained conversation models (PCMs) have demonstrated remarkable results in task-oriented dialogue (TOD) systems. Many PCMs focus predominantly on dialogue management tasks like dialogue state tracking, dialogue generation tasks like response generation, or both. However, the existing PCMs seldom consider dialogue comprehension tasks, such as dialogue question answering and summarization tasks. These tasks allow PCMs to glean dialogue context from various angles. This observation naturally raises the question: Can the performance of downstream dialogue tasks be enhanced if a PCM is pre-trained on dialogue management, generation, and comprehension tasks? To investigate this, we proposed an Omnipotent Dialogue pre-training model (OmniDialog). It unifies these three dialogue tasks into a monolithic framework by multi-task learning, fostering inter-task communication. The pre-training corpus of OmniDialog spans $\mathbf{7}$ dialogue-focused tasks, drawing from $\mathbf{15}$ datasets and encompassing over $\mathbf{3.2}$ million dialogue utterances. To our knowledge, OmniDialog is a pioneering PCM pre-trained across dialogue management, generation, and comprehension domains. We evaluated its performance across four tasks: dialogue summarization, end-to-end dialogue modeling, dialogue state tracking, and intent classification. The results underscore its efficacy in domain transfer learning, low-resource, and full-dataset scenarios. Furthermore, to glean a nuanced understanding of OmniDialog's strengths and potential pitfalls, we designed a fine-grained analysis framework for dialogue-centric tasks. Experimental results show that the OmniDialog is good at hard samples, such as long dialogues and lengthy responses.

Related papers

• Controllable Dialogue Simulation with In-Context Learning [39.04491297557292]
  textscDialogic is a dialogue simulation method based on large language model in-context learning. Our method can rapidly expand a small set of dialogue data with minimum or zero human involvement. Our simulated dialogues have near-human fluency and annotation accuracy.
  arXiv  Detail & Related papers  (2022-10-09T06:32:58Z)
• SPACE-3: Unified Dialog Model Pre-training for Task-Oriented Dialog Understanding and Generation [123.37377363355363]
  SPACE-3 is a novel unified semi-supervised pre-trained conversation model learning from large-scale dialog corpora. It can be effectively fine-tuned on a wide range of downstream dialog tasks. Results show that SPACE-3 achieves state-of-the-art performance on eight downstream dialog benchmarks.
  arXiv  Detail & Related papers  (2022-09-14T14:17:57Z)
• Act-Aware Slot-Value Predicting in Multi-Domain Dialogue State Tracking [5.816391291790977]
  Dialogue state tracking (DST) aims to track human-machine interactions and generate state representations for managing the dialogue. Recent advances in machine reading comprehension predict both categorical and non-categorical types of slots for dialogue state tracking. We formulate and incorporate dialogue acts, and leverage recent advances in machine reading comprehension to predict both categorical and non-categorical types of slots for dialogue state tracking.
  arXiv  Detail & Related papers  (2022-08-04T05:18:30Z)
• UniDS: A Unified Dialogue System for Chit-Chat and Task-oriented Dialogues [59.499965460525694]
  We propose a unified dialogue system (UniDS) with the two aforementioned skills. We design a unified dialogue data schema, compatible for both chit-chat and task-oriented dialogues. We train UniDS with mixed dialogue data from a pretrained chit-chat dialogue model.
  arXiv  Detail & Related papers  (2021-10-15T11:56:47Z)
• Distributed Structured Actor-Critic Reinforcement Learning for Universal Dialogue Management [29.57382819573169]
  We focus on devising a policy that chooses which dialogue action to respond to the user. The sequential system decision-making process can be abstracted into a partially observable Markov decision process. In the past few years, there are many deep reinforcement learning (DRL) algorithms, which use neural networks (NN) as function approximators.
  arXiv  Detail & Related papers  (2020-09-22T05:39:31Z)
• TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue [113.45485470103762]
  In this work, we unify nine human-human and multi-turn task-oriented dialogue datasets for language modeling. To better model dialogue behavior during pre-training, we incorporate user and system tokens into the masked language modeling.
  arXiv  Detail & Related papers  (2020-04-15T04:09:05Z)
• Conversation Learner -- A Machine Teaching Tool for Building Dialog Managers for Task-Oriented Dialog Systems [57.082447660944965]
  Conversation Learner is a machine teaching tool for building dialog managers. It enables dialog authors to create a dialog flow using familiar tools, converting the dialog flow into a parametric model. It allows dialog authors to improve the dialog manager over time by leveraging user-system dialog logs as training data.
  arXiv  Detail & Related papers  (2020-04-09T00:10:54Z)
• Masking Orchestration: Multi-task Pretraining for Multi-role Dialogue Representation Learning [50.5572111079898]
  Multi-role dialogue understanding comprises a wide range of diverse tasks such as question answering, act classification, dialogue summarization etc. While dialogue corpora are abundantly available, labeled data, for specific learning tasks, can be highly scarce and expensive. In this work, we investigate dialogue context representation learning with various types unsupervised pretraining tasks.
  arXiv  Detail & Related papers  (2020-02-27T04:36:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.