How to Choose How to Choose Your Chatbot: A Massively Multi-System MultiReference Data Set for Dialog Metric Evaluation

• URL: http://arxiv.org/abs/2305.14533v2
• Date: Tue, 19 Nov 2024 16:34:17 GMT
• Title: How to Choose How to Choose Your Chatbot: A Massively Multi-System MultiReference Data Set for Dialog Metric Evaluation
• Abstract summary: We release MMSMR, a Massively Multi-System MultiReference dataset to enable future work on metrics and evaluation for dialog. We create and release an 8-reference dialog dataset by extending single-reference evaluation sets. We then train 1750 systems and evaluate them on our novel test set and the DailyDialog dataset.
• Score: 7.781647951264202
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We release MMSMR, a Massively Multi-System MultiReference dataset to enable future work on metrics and evaluation for dialog. Automatic metrics for dialogue evaluation should be robust proxies for human judgments; however, the verification of robustness is currently far from satisfactory. To quantify the robustness correlation and understand what is necessary in a test set, we create and release an 8-reference dialog dataset by extending single-reference evaluation sets and introduce this new language learning conversation dataset. We then train 1750 systems and evaluate them on our novel test set and the DailyDialog dataset. We release the novel test set, and model hyper parameters, inference outputs, and metric scores for each system on a variety of datasets.

Related papers

• Dial HEALTHDIAL for Advice: A Multilingual and Multi-Parallel Spoken Dialogue Dataset for Knowledge-Grounded Information Seeking [72.66384771245389]
  This work introduces HEALTHDIAL, a large-scale, multilingual, and multi-parallel dataset for developing and evaluating retrieval-augmented generation (RAG)-based spoken dialogue systems.<n>The dataset comprises 6,000 information-seeking dialogues (1,500 per language) grounded in trusted content from the World Health Organization (WHO) and 163 hours of user speech recorded from native speakers of diverse dialects.
  arXiv  Detail & Related papers  (2026-05-28T15:47:09Z)
• Data Selection for Multi-turn Dialogue Instruction Tuning [36.29487117560294]
  textbfMDS (Multi-turn Dialogue Selection) is a dialogue-level framework that scores whole conversations rather than isolated turns.<n>MDS outperforms strong single-turn selectors, dialogue-level scorers, and baselines on three multi-turn benchmarks and an in-domain Banking test set.
  arXiv  Detail & Related papers  (2026-04-09T07:01:26Z)
• Commonsense Generation and Evaluation for Dialogue Systems using Large Language Models [8.556799193001341]
  This paper explores the task of performing turn-level data augmentation for dialogue system based on different types of commonsense relationships.<n>The proposed methodology takes advantage of the extended knowledge and zero-shot capabilities of pretrained Large Language Models (LLMs) to follow instructions.<n>Preliminary results suggest that our approach effectively harnesses LLMs capabilities for commonsense reasoning and evaluation in dialogue systems.
  arXiv  Detail & Related papers  (2025-06-24T10:18:05Z)
• Simple LLM Prompting is State-of-the-Art for Robust and Multilingual Dialogue Evaluation [7.767020408405403]
  We propose a novel framework that takes advantage of the strengths of current evaluation models with the newly-established paradigm of prompting Large Language Models (LLMs) Empirical results show our framework achieves state of the art results in terms of mean Spearman correlation scores across several benchmarks.
  arXiv  Detail & Related papers  (2023-08-31T15:19:28Z)
• XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languages [105.54207724678767]
  Data scarcity is a crucial issue for the development of highly multilingual NLP systems. We propose XTREME-UP, a benchmark defined by its focus on the scarce-data scenario rather than zero-shot. XTREME-UP evaluates the capabilities of language models across 88 under-represented languages over 9 key user-centric technologies.
  arXiv  Detail & Related papers  (2023-05-19T18:00:03Z)
• DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization [127.714919036388]
  DIONYSUS is a pre-trained encoder-decoder model for summarizing dialogues in any new domain. Our experiments show that DIONYSUS outperforms existing methods on six datasets.
  arXiv  Detail & Related papers  (2022-12-20T06:21:21Z)
• CGoDial: A Large-Scale Benchmark for Chinese Goal-oriented Dialog Evaluation [75.60156479374416]
  CGoDial is a new challenging and comprehensive Chinese benchmark for Goal-oriented Dialog evaluation. It contains 96,763 dialog sessions and 574,949 dialog turns totally, covering three datasets with different knowledge sources. To bridge the gap between academic benchmarks and spoken dialog scenarios, we either collect data from real conversations or add spoken features to existing datasets via crowd-sourcing.
  arXiv  Detail & Related papers  (2022-11-21T16:21:41Z)
• GODEL: Large-Scale Pre-Training for Goal-Directed Dialog [119.1397031992088]
  We introduce GODEL, a large pre-trained language model for dialog. We show that GODEL outperforms state-of-the-art pre-trained dialog models in few-shot fine-tuning setups. A novel feature of our evaluation methodology is the introduction of a notion of utility that assesses the usefulness of responses.
  arXiv  Detail & Related papers  (2022-06-22T18:19:32Z)
• What is wrong with you?: Leveraging User Sentiment for Automatic Dialog Evaluation [73.03318027164605]
  We propose to use information that can be automatically extracted from the next user utterance as a proxy to measure the quality of the previous system response. Our model generalizes across both spoken and written open-domain dialog corpora collected from real and paid users.
  arXiv  Detail & Related papers  (2022-03-25T22:09:52Z)
• Constructing Multi-Modal Dialogue Dataset by Replacing Text with Semantically Relevant Images [17.076424447172297]
  This paper proposes a 45k multi-modal dialogue dataset created with minimal human intervention. Our method to create such a dataset consists of (1) preparing and pre-processing text dialogue datasets, (2) creating image-mixed dialogues by using a text-to-image replacement technique, and (3) employing a contextual-similarity-based filtering step.
  arXiv  Detail & Related papers  (2021-07-19T08:44:11Z)
• A Comprehensive Assessment of Dialog Evaluation Metrics [9.34612743192798]
  Standard language evaluation metrics are ineffective for evaluating dialog. Recent research has proposed a number of novel, dialog-specific metrics that correlate better with human judgements. This paper provides a comprehensive assessment of recently proposed dialog evaluation metrics on a number of datasets.
  arXiv  Detail & Related papers  (2021-06-07T15:17:03Z)
• Dialog Simulation with Realistic Variations for Training Goal-Oriented Conversational Systems [14.206866126142002]
  Goal-oriented dialog systems enable users to complete specific goals like requesting information about a movie or booking a ticket. We propose an approach for automatically creating a large corpus of annotated dialogs from a few thoroughly annotated sample dialogs and the dialog schema. We achieve 18? 50% relative accuracy on a held-out test set compared to a baseline dialog generation approach.
  arXiv  Detail & Related papers  (2020-11-16T19:39:15Z)
• Learning an Unreferenced Metric for Online Dialogue Evaluation [53.38078951628143]
  We propose an unreferenced automated evaluation metric that uses large pre-trained language models to extract latent representations of utterances. We show that our model achieves higher correlation with human annotations in an online setting, while not requiring true responses for comparison during inference.
  arXiv  Detail & Related papers  (2020-05-01T20:01:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.