A Knowledge-Grounded Dialog System Based on Pre-Trained Language Models

• URL: http://arxiv.org/abs/2106.14444v1
• Date: Mon, 28 Jun 2021 07:56:10 GMT
• Title: A Knowledge-Grounded Dialog System Based on Pre-Trained Language Models
• Authors: Weijie Zhang, Jiaoxuan Chen, Haipang Wu, Sanhui Wan, Gongfeng Li
• Abstract summary: We present a knowledge-grounded dialog system developed for the ninth Dialog System Technology Challenge (DSTC9) We leverage transfer learning with existing language models to accomplish the tasks in this challenge track.
• Score: 0.7699714865575189
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present a knowledge-grounded dialog system developed for the ninth Dialog System Technology Challenge (DSTC9) Track 1 - Beyond Domain APIs: Task-oriented Conversational Modeling with Unstructured Knowledge Access. We leverage transfer learning with existing language models to accomplish the tasks in this challenge track. Specifically, we divided the task into four sub-tasks and fine-tuned several Transformer models on each of the sub-tasks. We made additional changes that yielded gains in both performance and efficiency, including the combination of the model with traditional entity-matching techniques, and the addition of a pointer network to the output layer of the language model.

Related papers

• VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translation [91.39949385661379]
  VioLA is a single auto-regressive Transformer decoder-only network that unifies various cross-modal tasks involving speech and text. We first convert all the speech utterances to discrete tokens using an offline neural encoder. We further integrate task IDs (TID) and language IDs (LID) into the proposed model to enhance the modeling capability of handling different languages and tasks.
  arXiv  Detail & Related papers  (2023-05-25T14:39:47Z)
• DialogZoo: Large-Scale Dialog-Oriented Task Learning [52.18193690394549]
  We aim to build a unified foundation model which can solve massive diverse dialogue tasks. To achieve this goal, we first collect a large-scale well-labeled dialogue dataset from 73 publicly available datasets.
  arXiv  Detail & Related papers  (2022-05-25T11:17:16Z)
• A Study on Prompt-based Few-Shot Learning Methods for Belief State Tracking in Task-oriented Dialog Systems [10.024834304960846]
  We tackle the Dialogue Belief State Tracking problem of task-oriented conversational systems. Recent approaches to this problem leveraging Transformer-based models have yielded great results. We explore prompt-based few-shot learning for Dialogue Belief State Tracking.
  arXiv  Detail & Related papers  (2022-04-18T05:29:54Z)
• Towards Generalized Models for Task-oriented Dialogue Modeling on Spoken Conversations [22.894541507068933]
  This paper presents our approach to build generalized models for the Knowledge-grounded Task-oriented Dialogue Modeling on Spoken Conversations Challenge of DSTC-10. We employ extensive data augmentation strategies on written data, including artificial error injection and round-trip text-speech transformation. Our approach ranks third on the objective evaluation and second on the final official human evaluation.
  arXiv  Detail & Related papers  (2022-03-08T12:26:57Z)
• GKS: Graph-based Knowledge Selector for Task-oriented Dialog System [0.688204255655161]
  Graph-Knowledge Selector (GKS) outperforms SOTA models proposed in the data-set on knowledge selection from the 9th Dialog System Technology Challenges (DSTC9) GKS makes knowledge selection decisions in the dialog by simultaneously considering each knowledge embedding generated from the language model, without sequential features.
  arXiv  Detail & Related papers  (2021-12-07T14:16:26Z)
• Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue System [26.837972034630003]
  PPTOD is a unified plug-and-play model for task-oriented dialogue. We extensively test our model on three benchmark TOD tasks, including end-to-end dialogue modelling, dialogue state tracking, and intent classification.
  arXiv  Detail & Related papers  (2021-09-29T22:02:18Z)
• Overview of the Ninth Dialog System Technology Challenge: DSTC9 [111.35889309106359]
  The Ninth Dialog System Technology Challenge (DSTC-9) focuses on applying end-to-end dialog technologies for four distinct tasks in dialog systems. This paper describes the task definition, provided datasets, baselines and evaluation set-up for each track.
  arXiv  Detail & Related papers  (2020-11-12T16:43:10Z)
• On Task-Level Dialogue Composition of Generative Transformer Model [9.751234480029765]
  We study the effect of training human-human task-oriented dialogues towards improving the ability to compose multiple tasks on Transformer generative models. To that end, we propose and explore two solutions: (1) creating synthetic multiple task dialogue data for training from human-human single task dialogue and (2) forcing the encoder representation to be invariant to single and multiple task dialogues using an auxiliary loss.
  arXiv  Detail & Related papers  (2020-10-09T22:10:03Z)
• Video-Grounded Dialogues with Pretrained Generation Language Models [88.15419265622748]
  We leverage the power of pre-trained language models for improving video-grounded dialogue. We propose a framework by formulating sequence-to-grounded dialogue tasks as a sequence-to-grounded task. Our framework allows fine-tuning language models to capture dependencies across multiple modalities.
  arXiv  Detail & Related papers  (2020-06-27T08:24:26Z)
• SOLOIST: Building Task Bots at Scale with Transfer Learning and Machine Teaching [81.45928589522032]
  We parameterize modular task-oriented dialog systems using a Transformer-based auto-regressive language model. We pre-train, on heterogeneous dialog corpora, a task-grounded response generation model. Experiments show that SOLOIST creates new state-of-the-art on well-studied task-oriented dialog benchmarks.
  arXiv  Detail & Related papers  (2020-05-11T17:58:34Z)
• TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue [113.45485470103762]
  In this work, we unify nine human-human and multi-turn task-oriented dialogue datasets for language modeling. To better model dialogue behavior during pre-training, we incorporate user and system tokens into the masked language modeling.
  arXiv  Detail & Related papers  (2020-04-15T04:09:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.