Representation Learning for Conversational Data using Discourse Mutual Information Maximization

• URL: http://arxiv.org/abs/2112.05787v1
• Date: Sat, 4 Dec 2021 13:17:07 GMT
• Title: Representation Learning for Conversational Data using Discourse Mutual Information Maximization
• Authors: Bishal Santra, Sumegh Roychowdhury, Aishik Mandal, Vasu Gurram, Atharva Naik, Manish Gupta, Pawan Goyal
• Abstract summary: We argue that the structure-unaware word-by-word generation is not suitable for effective conversation modeling. We propose a structure-aware Mutual Information based loss-function DMI for training dialog-representation models. Our models show the most promising performance on the dialog evaluation task DailyDialog++, in both random and adversarial negative scenarios.
• Score: 9.017156603976915
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Although many pretrained models exist for text or images, there have been relatively fewer attempts to train representations specifically for dialog understanding. Prior works usually relied on finetuned representations based on generic text representation models like BERT or GPT-2. But, existing pretraining objectives do not take the structural information of text into consideration. Although generative dialog models can learn structural features too, we argue that the structure-unaware word-by-word generation is not suitable for effective conversation modeling. We empirically demonstrate that such representations do not perform consistently across various dialog understanding tasks. Hence, we propose a structure-aware Mutual Information based loss-function DMI (Discourse Mutual Information) for training dialog-representation models, that additionally captures the inherent uncertainty in response prediction. Extensive evaluation on nine diverse dialog modeling tasks shows that our proposed DMI-based models outperform strong baselines by significant margins, even with small-scale pretraining. Our models show the most promising performance on the dialog evaluation task DailyDialog++, in both random and adversarial negative scenarios.

Related papers

• DialCLIP: Empowering CLIP as Multi-Modal Dialog Retriever [83.33209603041013]
  We propose a parameter-efficient prompt-tuning method named DialCLIP for multi-modal dialog retrieval. Our approach introduces a multi-modal context generator to learn context features which are distilled into prompts within the pre-trained vision-language model CLIP. To facilitate various types of retrieval, we also design multiple experts to learn mappings from CLIP outputs to multi-modal representation space.
  arXiv  Detail & Related papers  (2024-01-02T07:40:12Z)
• Pre-training Multi-party Dialogue Models with Latent Discourse Inference [85.9683181507206]
  We pre-train a model that understands the discourse structure of multi-party dialogues, namely, to whom each utterance is replying. To fully utilize the unlabeled data, we propose to treat the discourse structures as latent variables, then jointly infer them and pre-train the discourse-aware model.
  arXiv  Detail & Related papers  (2023-05-24T14:06:27Z)
• Stabilized In-Context Learning with Pre-trained Language Models for Few Shot Dialogue State Tracking [57.92608483099916]
  Large pre-trained language models (PLMs) have shown impressive unaided performance across many NLP tasks. For more complex tasks such as dialogue state tracking (DST), designing prompts that reliably convey the desired intent is nontrivial. We introduce a saliency model to limit dialogue text length, allowing us to include more exemplars per query.
  arXiv  Detail & Related papers  (2023-02-12T15:05:10Z)
• SPACE-2: Tree-Structured Semi-Supervised Contrastive Pre-training for Task-Oriented Dialog Understanding [68.94808536012371]
  We propose a tree-structured pre-trained conversation model, which learns dialog representations from limited labeled dialogs and large-scale unlabeled dialog corpora. Our method can achieve new state-of-the-art results on the DialoGLUE benchmark consisting of seven datasets and four popular dialog understanding tasks.
  arXiv  Detail & Related papers  (2022-09-14T13:42:50Z)
• GODEL: Large-Scale Pre-Training for Goal-Directed Dialog [119.1397031992088]
  We introduce GODEL, a large pre-trained language model for dialog. We show that GODEL outperforms state-of-the-art pre-trained dialog models in few-shot fine-tuning setups. A novel feature of our evaluation methodology is the introduction of a notion of utility that assesses the usefulness of responses.
  arXiv  Detail & Related papers  (2022-06-22T18:19:32Z)
• Modeling Coreference Relations in Visual Dialog [18.926582410644375]
  The occurrences of coreference relations in the dialog makes it a more challenging task than visual question-answering. We propose two soft constraints that can improve the model's ability of resolving coreferences in dialog in an unsupervised way.
  arXiv  Detail & Related papers  (2022-03-06T15:22:24Z)
• Response Generation with Context-Aware Prompt Learning [19.340498579331555]
  We present a novel approach for pre-trained dialogue modeling that casts the dialogue generation problem as a prompt-learning task. Instead of fine-tuning on limited dialogue data, our approach, DialogPrompt, learns continuous prompt embeddings optimized for dialogue contexts. Our approach significantly outperforms the fine-tuning baseline and the generic prompt-learning methods.
  arXiv  Detail & Related papers  (2021-11-04T05:40:13Z)
• Self-training Improves Pre-training for Few-shot Learning in Task-oriented Dialog Systems [47.937191088981436]
  Large-scale pre-trained language models, have shown promising results for few-shot learning in ToD. We propose a self-training approach that iteratively labels the most confident unlabeled data to train a stronger Student model. We conduct experiments and present analyses on four downstream tasks in ToD, including intent classification, dialog state tracking, dialog act prediction, and response selection.
  arXiv  Detail & Related papers  (2021-08-28T07:22:06Z)
• RADDLE: An Evaluation Benchmark and Analysis Platform for Robust Task-oriented Dialog Systems [75.87418236410296]
  We introduce the RADDLE benchmark, a collection of corpora and tools for evaluating the performance of models across a diverse set of domains. RADDLE is designed to favor and encourage models with a strong generalization ability. We evaluate recent state-of-the-art systems based on pre-training and fine-tuning, and find that grounded pre-training on heterogeneous dialog corpora performs better than training a separate model per domain.
  arXiv  Detail & Related papers  (2020-12-29T08:58:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.