The Lab vs The Crowd: An Investigation into Data Quality for Neural Dialogue Models

• URL: http://arxiv.org/abs/2012.03855v1
• Date: Mon, 7 Dec 2020 17:02:00 GMT
• Title: The Lab vs The Crowd: An Investigation into Data Quality for Neural Dialogue Models
• Authors: Jos\'e Lopes, Francisco J. Chiyah Garcia and Helen Hastie
• Abstract summary: We compare the performance of dialogue models for the same interaction task but collected in two settings: in the lab vs. crowd-sourced. We find that fewer lab dialogues are needed to reach similar accuracy, less than half the amount of lab data as crowd-sourced data.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Challenges around collecting and processing quality data have hampered progress in data-driven dialogue models. Previous approaches are moving away from costly, resource-intensive lab settings, where collection is slow but where the data is deemed of high quality. The advent of crowd-sourcing platforms, such as Amazon Mechanical Turk, has provided researchers with an alternative cost-effective and rapid way to collect data. However, the collection of fluid, natural spoken or textual interaction can be challenging, particularly between two crowd-sourced workers. In this study, we compare the performance of dialogue models for the same interaction task but collected in two different settings: in the lab vs. crowd-sourced. We find that fewer lab dialogues are needed to reach similar accuracy, less than half the amount of lab data as crowd-sourced data. We discuss the advantages and disadvantages of each data collection method.

Related papers

• Deep Active Learning for Data Mining from Conflict Text Corpora [0.0]
  This paper proposes one such approach that is inexpensive and high performance, leveraging active learning. The approach shows performance similar to human (gold-standard) coding while reducing the amount of required human annotation by as much as 99%.
  arXiv  Detail & Related papers  (2024-02-02T17:16:23Z)
• AutoConv: Automatically Generating Information-seeking Conversations with Large Language Models [74.10293412011455]
  We propose AutoConv for synthetic conversation generation. Specifically, we formulate the conversation generation problem as a language modeling task. We finetune an LLM with a few human conversations to capture the characteristics of the information-seeking process.
  arXiv  Detail & Related papers  (2023-08-12T08:52:40Z)
• Does Collaborative Human-LM Dialogue Generation Help Information Extraction from Human Dialogues? [55.28340832822234]
  Problem-solving human dialogues in real applications can be much more complex than existing Wizard-of-Oz collections. We introduce a human-in-the-loop dialogue generation framework capable of synthesizing realistic dialogues.
  arXiv  Detail & Related papers  (2023-07-13T20:02:50Z)
• A Model-Agnostic Data Manipulation Method for Persona-based Dialogue Generation [107.82729587882397]
  It is expensive to scale up current persona-based dialogue datasets. Each data sample in this task is more complex to learn with than conventional dialogue data. We propose a data manipulation method, which is model-agnostic to be packed with any persona-based dialogue generation model.
  arXiv  Detail & Related papers  (2022-04-21T03:49:54Z)
• Human-in-the-Loop for Data Collection: a Multi-Target Counter Narrative Dataset to Fight Online Hate Speech [10.323063834827416]
  Undermining the impact of hateful content with informed and non-aggressive responses, called counter narratives, has emerged as a possible solution for having healthier online communities. We propose a novel human-in-the-loop data collection methodology in which a generative language model is refined iteratively. Results show that the methodology is scalable and facilitates diverse, novel, and cost-effective data collection.
  arXiv  Detail & Related papers  (2021-07-19T09:45:54Z)
• What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks? [31.39009622826369]
  We compare the efficacy of interventions that have been proposed in prior work as ways of improving data quality. We find that asking workers to write explanations for their examples is an ineffective stand-alone strategy for boosting NLU example difficulty. We observe that the data from the iterative protocol with expert assessments is more challenging by several measures.
  arXiv  Detail & Related papers  (2021-06-01T21:05:52Z)
• Learning to Match Jobs with Resumes from Sparse Interaction Data using Multi-View Co-Teaching Network [83.64416937454801]
  Job-resume interaction data is sparse and noisy, which affects the performance of job-resume match algorithms. We propose a novel multi-view co-teaching network from sparse interaction data for job-resume matching. Our model is able to outperform state-of-the-art methods for job-resume matching.
  arXiv  Detail & Related papers  (2020-09-25T03:09:54Z)
• Dialogue Distillation: Open-Domain Dialogue Augmentation Using Unpaired Data [61.71319905364992]
  We propose a novel data augmentation method for training open-domain dialogue models by utilizing unpaired data. A data-level distillation process is first proposed to construct augmented dialogues where both post and response are retrieved from the unpaired data. A ranking module is employed to filter out low-quality dialogues. A model-level distillation process is employed to distill a teacher model trained on high-quality paired data to augmented dialogue pairs.
  arXiv  Detail & Related papers  (2020-09-20T13:06:38Z)
• Improving Multi-Turn Response Selection Models with Complementary Last-Utterance Selection by Instance Weighting [84.9716460244444]
  We consider utilizing the underlying correlation in the data resource itself to derive different kinds of supervision signals. We conduct extensive experiments in two public datasets and obtain significant improvement in both datasets.
  arXiv  Detail & Related papers  (2020-02-18T06:29:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.