Masking Orchestration: Multi-task Pretraining for Multi-role Dialogue Representation Learning

• URL: http://arxiv.org/abs/2003.04994v1
• Date: Thu, 27 Feb 2020 04:36:52 GMT
• Title: Masking Orchestration: Multi-task Pretraining for Multi-role Dialogue Representation Learning
• Authors: Tianyi Wang, Yating Zhang, Xiaozhong Liu, Changlong Sun, Qiong Zhang
• Abstract summary: Multi-role dialogue understanding comprises a wide range of diverse tasks such as question answering, act classification, dialogue summarization etc. While dialogue corpora are abundantly available, labeled data, for specific learning tasks, can be highly scarce and expensive. In this work, we investigate dialogue context representation learning with various types unsupervised pretraining tasks.
• Score: 50.5572111079898
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multi-role dialogue understanding comprises a wide range of diverse tasks such as question answering, act classification, dialogue summarization etc. While dialogue corpora are abundantly available, labeled data, for specific learning tasks, can be highly scarce and expensive. In this work, we investigate dialogue context representation learning with various types unsupervised pretraining tasks where the training objectives are given naturally according to the nature of the utterance and the structure of the multi-role conversation. Meanwhile, in order to locate essential information for dialogue summarization/extraction, the pretraining process enables external knowledge integration. The proposed fine-tuned pretraining mechanism is comprehensively evaluated via three different dialogue datasets along with a number of downstream dialogue-mining tasks. Result shows that the proposed pretraining mechanism significantly contributes to all the downstream tasks without discrimination to different encoders.

Related papers

• Self-Explanation Prompting Improves Dialogue Understanding in Large Language Models [52.24756457516834]
  We propose a novel "Self-Explanation" prompting strategy to enhance the comprehension abilities of Large Language Models (LLMs) This task-agnostic approach requires the model to analyze each dialogue utterance before task execution, thereby improving performance across various dialogue-centric tasks. Experimental results from six benchmark datasets confirm that our method consistently outperforms other zero-shot prompts and matches or exceeds the efficacy of few-shot prompts.
  arXiv  Detail & Related papers  (2023-09-22T15:41:34Z)
• Revisiting Conversation Discourse for Dialogue Disentanglement [88.3386821205896]
  We propose enhancing dialogue disentanglement by taking full advantage of the dialogue discourse characteristics. We develop a structure-aware framework to integrate the rich structural features for better modeling the conversational semantic context. Our work has great potential to facilitate broader multi-party multi-thread dialogue applications.
  arXiv  Detail & Related papers  (2023-06-06T19:17:47Z)
• PaCE: Unified Multi-modal Dialogue Pre-training with Progressive and Compositional Experts [45.69829921539995]
  This paper proposes textbfPaCE, a unified, structured, compositional multi-modal dialogue pre-training framework. It utilizes a combination of several fundamental experts to accommodate multiple dialogue-related tasks and can be pre-trained using limited dialogue and extensive non-dialogue multi-modal data. Experimental results demonstrate that PaCE achieves state-of-the-art results on eight multi-modal dialog benchmarks.
  arXiv  Detail & Related papers  (2023-05-24T07:43:29Z)
• Advances in Multi-turn Dialogue Comprehension: A Survey [51.215629336320305]
  Training machines to understand natural language and interact with humans is an elusive and essential task of artificial intelligence. This paper reviews the previous methods from the technical perspective of dialogue modeling for the dialogue comprehension task. In addition, we categorize dialogue-related pre-training techniques which are employed to enhance PrLMs in dialogue scenarios.
  arXiv  Detail & Related papers  (2021-10-11T03:52:37Z)
• Dialogue-oriented Pre-training [70.03028879331339]
  We propose three strategies to simulate the conversation features on general plain text. Dialog-PrLM is fine-tuned on three public multi-turn dialogue datasets.
  arXiv  Detail & Related papers  (2021-06-01T12:02:46Z)
• Advances in Multi-turn Dialogue Comprehension: A Survey [51.215629336320305]
  We review the previous methods from the perspective of dialogue modeling. We discuss three typical patterns of dialogue modeling that are widely-used in dialogue comprehension tasks.
  arXiv  Detail & Related papers  (2021-03-04T15:50:17Z)
• MultiTalk: A Highly-Branching Dialog Testbed for Diverse Conversations [39.81965687032923]
  We present the MultiTalk dataset, a corpus of over 320,000 sentences of written conversational dialog. We make multiple contributions to study dialog generation in the highly branching setting. Our culminating task is a challenging theory of mind problem, a controllable generation task.
  arXiv  Detail & Related papers  (2021-02-02T02:29:40Z)
• Utterance-level Dialogue Understanding: An Empirical Study [43.35258958775454]
  This paper explores and quantify the role of context for different aspects of a dialogue. Specifically, we employ various perturbations to distort the context of a given utterance. This provides us with insights into the fundamental contextual controlling factors of different aspects of a dialogue.
  arXiv  Detail & Related papers  (2020-09-29T09:50:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.