Interactive Evaluation of Dialog Track at DSTC9

• URL: http://arxiv.org/abs/2207.14403v1
• Date: Thu, 28 Jul 2022 22:54:04 GMT
• Title: Interactive Evaluation of Dialog Track at DSTC9
• Authors: Shikib Mehri, Yulan Feng, Carla Gordon, Seyed Hossein Alavi, David Traum, Maxine Eskenazi
• Abstract summary: The Interactive Evaluation of Dialog Track was introduced at the 9th Dialog System Technology Challenge. This paper provides an overview of the track, including the methodology and results.
• Score: 8.2208199207543
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The ultimate goal of dialog research is to develop systems that can be effectively used in interactive settings by real users. To this end, we introduced the Interactive Evaluation of Dialog Track at the 9th Dialog System Technology Challenge. This track consisted of two sub-tasks. The first sub-task involved building knowledge-grounded response generation models. The second sub-task aimed to extend dialog models beyond static datasets by assessing them in an interactive setting with real users. Our track challenges participants to develop strong response generation models and explore strategies that extend them to back-and-forth interactions with real users. The progression from static corpora to interactive evaluation introduces unique challenges and facilitates a more thorough assessment of open-domain dialog systems. This paper provides an overview of the track, including the methodology and results. Furthermore, it provides insights into how to best evaluate open-domain dialog models

Related papers

• Dialogue Agents 101: A Beginner's Guide to Critical Ingredients for Designing Effective Conversational Systems [29.394466123216258]
  This study provides a comprehensive overview of the primary characteristics of a dialogue agent, their corresponding open-domain datasets, and the methods used to benchmark these datasets. We propose UNIT, a UNified dIalogue dataseT constructed from conversations of existing datasets for different dialogue tasks capturing the nuances for each of them.
  arXiv  Detail & Related papers  (2023-07-14T10:05:47Z)
• Is MultiWOZ a Solved Task? An Interactive TOD Evaluation Framework with User Simulator [37.590563896382456]
  We propose an interactive evaluation framework for Task-Oriented Dialogue (TOD) systems. We first build a goal-oriented user simulator based on pre-trained models and then use the user simulator to interact with the dialogue system to generate dialogues. Experimental results show that RL-based TOD systems trained by our proposed user simulator can achieve nearly 98% inform and success rates.
  arXiv  Detail & Related papers  (2022-10-26T07:41:32Z)
• Manual-Guided Dialogue for Flexible Conversational Agents [84.46598430403886]
  How to build and use dialogue data efficiently, and how to deploy models in different domains at scale can be critical issues in building a task-oriented dialogue system. We propose a novel manual-guided dialogue scheme, where the agent learns the tasks from both dialogue and manuals. Our proposed scheme reduces the dependence of dialogue models on fine-grained domain ontology, and makes them more flexible to adapt to various domains.
  arXiv  Detail & Related papers  (2022-08-16T08:21:12Z)
• GODEL: Large-Scale Pre-Training for Goal-Directed Dialog [119.1397031992088]
  We introduce GODEL, a large pre-trained language model for dialog. We show that GODEL outperforms state-of-the-art pre-trained dialog models in few-shot fine-tuning setups. A novel feature of our evaluation methodology is the introduction of a notion of utility that assesses the usefulness of responses.
  arXiv  Detail & Related papers  (2022-06-22T18:19:32Z)
• Target-Guided Dialogue Response Generation Using Commonsense and Data Augmentation [32.764356638437214]
  We introduce a new technique for target-guided response generation. We also propose techniques to re-purpose existing dialogue datasets for target-guided generation. Our work generally enables dialogue system designers to exercise more control over the conversations that their systems produce.
  arXiv  Detail & Related papers  (2022-05-19T04:01:40Z)
• KETOD: Knowledge-Enriched Task-Oriented Dialogue [77.59814785157877]
  Existing studies in dialogue system research mostly treat task-oriented dialogue and chit-chat as separate domains. We investigate how task-oriented dialogue and knowledge-grounded chit-chat can be effectively integrated into a single model.
  arXiv  Detail & Related papers  (2022-05-11T16:01:03Z)
• A Unified Pre-training Framework for Conversational AI [25.514505462661763]
  PLATO-2 is trained via two-stage curriculum learning to fit the simplified one-to-one mapping relationship. PLATO-2 has obtained the 1st place in all three tasks, verifying its effectiveness as a unified framework for various dialogue systems.
  arXiv  Detail & Related papers  (2021-05-06T07:27:11Z)
• Overview of the Ninth Dialog System Technology Challenge: DSTC9 [111.35889309106359]
  The Ninth Dialog System Technology Challenge (DSTC-9) focuses on applying end-to-end dialog technologies for four distinct tasks in dialog systems. This paper describes the task definition, provided datasets, baselines and evaluation set-up for each track.
  arXiv  Detail & Related papers  (2020-11-12T16:43:10Z)
• Probing Neural Dialog Models for Conversational Understanding [21.76744391202041]
  We analyze the internal representations learned by neural open-domain dialog systems. Our results suggest that standard open-domain dialog systems struggle with answering questions. We also find that the dyadic, turn-taking nature of dialog is not fully leveraged by these models.
  arXiv  Detail & Related papers  (2020-06-07T17:32:00Z)
• Recent Advances and Challenges in Task-oriented Dialog System [63.82055978899631]
  Task-oriented dialog systems are attracting more and more attention in academic and industrial communities. We discuss three critical topics for task-oriented dialog systems: (1) improving data efficiency to facilitate dialog modeling in low-resource settings, (2) modeling multi-turn dynamics for dialog policy learning, and (3) integrating domain knowledge into the dialog model.
  arXiv  Detail & Related papers  (2020-03-17T01:34:56Z)
• Variational Hierarchical Dialog Autoencoder for Dialog State Tracking Data Augmentation [59.174903564894954]
  In this work, we extend this approach to the task of dialog state tracking for goal-oriented dialogs. We propose the Variational Hierarchical Dialog Autoencoder (VHDA) for modeling the complete aspects of goal-oriented dialogs. Experiments on various dialog datasets show that our model improves the downstream dialog trackers' robustness via generative data augmentation.
  arXiv  Detail & Related papers  (2020-01-23T15:34:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.