ComperDial: Commonsense Persona-grounded Dialogue Dataset and Benchmark

• URL: http://arxiv.org/abs/2406.11228v1
• Date: Mon, 17 Jun 2024 05:51:04 GMT
• Title: ComperDial: Commonsense Persona-grounded Dialogue Dataset and Benchmark
• Authors: Hiromi Wakaki, Yuki Mitsufuji, Yoshinori Maeda, Yukiko Nishimura, Silin Gao, Mengjie Zhao, Keiichi Yamada, Antoine Bosselut,
• Abstract summary: ComperDial consists of human-scored responses for 10,395 dialogue turns in 1,485 conversations collected from 99 dialogue agents. In addition to single-turn response scores, ComperDial also contains dialogue-level human-annotated scores. Building off ComperDial, we devise a new automatic evaluation metric to measure the general similarity of model-generated dialogues to human conversations.
• Score: 26.100299485985197
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose a new benchmark, ComperDial, which facilitates the training and evaluation of evaluation metrics for open-domain dialogue systems. ComperDial consists of human-scored responses for 10,395 dialogue turns in 1,485 conversations collected from 99 dialogue agents submitted to the Commonsense Persona-grounded Dialogue (CPD) challenge. As a result, for any dialogue, our benchmark includes multiple diverse responses with variety of characteristics to ensure more robust evaluation of learned dialogue metrics. In addition to single-turn response scores, ComperDial also contains dialogue-level human-annotated scores, enabling joint assessment of multi-turn model responses throughout a dialogue. Finally, building off ComperDial, we devise a new automatic evaluation metric to measure the general similarity of model-generated dialogues to human conversations. Our experimental results demonstrate that our novel metric, CPDScore is more correlated with human judgments than existing metrics. We release both ComperDial and CPDScore to the community to accelerate development of automatic evaluation metrics for open-domain dialogue systems.

Related papers

• PairEval: Open-domain Dialogue Evaluation with Pairwise Comparison [38.03304773600225]
  PairEval is a novel dialogue evaluation metric for assessing responses by comparing their quality against responses in different conversations. We show that PairEval exhibits a higher correlation with human judgments than baseline metrics. We also find that the proposed comparative metric is more robust in detecting common failures from open-domain dialogue systems.
  arXiv  Detail & Related papers  (2024-04-01T09:35:06Z)
• DiQAD: A Benchmark Dataset for End-to-End Open-domain Dialogue Assessment [38.26039323208791]
  We release a large-scale dialogue quality assessment dataset (DiQAD) for automatically assessing open-domain dialogue quality. Specifically, we establish the assessment criteria based on the dimensions conforming to human judgements on dialogue qualities. We also annotate large-scale dialogues that conversed between real users based on these annotation criteria, which contains around 100,000 dialogues.
  arXiv  Detail & Related papers  (2023-10-25T03:04:57Z)
• Toward More Accurate and Generalizable Evaluation Metrics for Task-Oriented Dialogs [19.43845920149182]
  We introduce a new dialog-level annotation workflow called Dialog Quality. DQA expert annotators evaluate the quality of dialogs as a whole, and also label dialogs for attributes such as goal completion and user sentiment. We argue that having high-quality human-annotated data is an important component of evaluating interaction quality for large industrial-scale voice assistant platforms.
  arXiv  Detail & Related papers  (2023-06-06T19:43:29Z)
• FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluation [58.46761798403072]
  We propose a dialogue-level metric that consists of three sub-metrics with each targeting a specific dimension. The sub-metrics are trained with novel self-supervised objectives and exhibit strong correlations with human judgment for their respective dimensions. Compared to the existing state-of-the-art metric, the combined metrics achieve around 16% relative improvement on average.
  arXiv  Detail & Related papers  (2022-10-25T08:26:03Z)
• MDD-Eval: Self-Training on Augmented Data for Multi-Domain Dialogue Evaluation [66.60285024216573]
  A dialogue evaluator is expected to conduct assessment across domains as well. Most of the state-of-the-art automatic dialogue evaluation metrics (ADMs) are not designed for multi-domain evaluation. We are motivated to design a general and robust framework, MDD-Eval, to address the problem.
  arXiv  Detail & Related papers  (2021-12-14T07:01:20Z)
• A Comprehensive Assessment of Dialog Evaluation Metrics [9.34612743192798]
  Standard language evaluation metrics are ineffective for evaluating dialog. Recent research has proposed a number of novel, dialog-specific metrics that correlate better with human judgements. This paper provides a comprehensive assessment of recently proposed dialog evaluation metrics on a number of datasets.
  arXiv  Detail & Related papers  (2021-06-07T15:17:03Z)
• DynaEval: Unifying Turn and Dialogue Level Evaluation [60.66883575106898]
  We propose DynaEval, a unified automatic evaluation framework. It is capable of performing turn-level evaluation, but also holistically considers the quality of the entire dialogue. Experiments show that DynaEval significantly outperforms the state-of-the-art dialogue coherence model.
  arXiv  Detail & Related papers  (2021-06-02T12:23:18Z)
• Assessing Dialogue Systems with Distribution Distances [48.61159795472962]
  We propose to measure the performance of a dialogue system by computing the distribution-wise distance between its generated conversations and real-world conversations. Experiments on several dialogue corpora show that our proposed metrics correlate better with human judgments than existing metrics.
  arXiv  Detail & Related papers  (2021-05-06T10:30:13Z)
• Is this Dialogue Coherent? Learning from Dialogue Acts and Entities [82.44143808977209]
  We create the Switchboard Coherence (SWBD-Coh) corpus, a dataset of human-human spoken dialogues annotated with turn coherence ratings. Our statistical analysis of the corpus indicates how turn coherence perception is affected by patterns of distribution of entities. We find that models combining both DA and entity information yield the best performances both for response selection and turn coherence rating.
  arXiv  Detail & Related papers  (2020-06-17T21:02:40Z)
• Is Your Goal-Oriented Dialog Model Performing Really Well? Empirical Analysis of System-wise Evaluation [114.48767388174218]
  This paper presents an empirical analysis on different types of dialog systems composed of different modules in different settings. Our results show that a pipeline dialog system trained using fine-grained supervision signals at different component levels often obtains better performance than the systems that use joint or end-to-end models trained on coarse-grained labels.
  arXiv  Detail & Related papers  (2020-05-15T05:20:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.