Design Choices for Crowdsourcing Implicit Discourse Relations: Revealing the Biases Introduced by Task Design

• URL: http://arxiv.org/abs/2304.00815v1
• Date: Mon, 3 Apr 2023 09:04:18 GMT
• Title: Design Choices for Crowdsourcing Implicit Discourse Relations: Revealing the Biases Introduced by Task Design
• Authors: Valentina Pyatkin, Frances Yung, Merel C.J. Scholman, Reut Tsarfaty, Ido Dagan, Vera Demberg
• Abstract summary: We show that the task design can push annotators towards certain relations. We conclude that this type of bias should be taken into account when training and testing models.
• Score: 23.632204469647526
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Disagreement in natural language annotation has mostly been studied from a perspective of biases introduced by the annotators and the annotation frameworks. Here, we propose to analyze another source of bias: task design bias, which has a particularly strong impact on crowdsourced linguistic annotations where natural language is used to elicit the interpretation of laymen annotators. For this purpose we look at implicit discourse relation annotation, a task that has repeatedly been shown to be difficult due to the relations' ambiguity. We compare the annotations of 1,200 discourse relations obtained using two distinct annotation tasks and quantify the biases of both methods across four different domains. Both methods are natural language annotation tasks designed for crowdsourcing. We show that the task design can push annotators towards certain relations and that some discourse relations senses can be better elicited with one or the other annotation approach. We also conclude that this type of bias should be taken into account when training and testing models.

Related papers

• Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval [109.62363167257664]
  We propose a generative model for learning multilingual text embeddings. Our model operates on parallel data in $N$ languages. We evaluate this method on a suite of tasks including semantic similarity, bitext mining, and cross-lingual question retrieval.
  arXiv  Detail & Related papers  (2022-12-21T02:41:40Z)
• Challenges in Measuring Bias via Open-Ended Language Generation [1.5552869983952944]
  We analyze how specific choices of prompt sets, metrics, automatic tools and sampling strategies affect bias results. We provide recommendations for reporting biases in open-ended language generation.
  arXiv  Detail & Related papers  (2022-05-23T19:57:15Z)
• A Latent-Variable Model for Intrinsic Probing [93.62808331764072]
  We propose a novel latent-variable formulation for constructing intrinsic probes. We find empirical evidence that pre-trained representations develop a cross-lingually entangled notion of morphosyntax.
  arXiv  Detail & Related papers  (2022-01-20T15:01:12Z)
• Two Contrasting Data Annotation Paradigms for Subjective NLP Tasks [17.033055327465238]
  We propose two contrasting paradigms for data annotation. The descriptive paradigm encourages annotator subjectivity, whereas the prescriptive paradigm discourages it. We argue that dataset creators should explicitly aim for one or the other to facilitate the intended use of their dataset.
  arXiv  Detail & Related papers  (2021-12-14T15:38:22Z)
• Annotation Curricula to Implicitly Train Non-Expert Annotators [56.67768938052715]
  voluntary studies often require annotators to familiarize themselves with the task, its annotation scheme, and the data domain. This can be overwhelming in the beginning, mentally taxing, and induce errors into the resulting annotations. We propose annotation curricula, a novel approach to implicitly train annotators.
  arXiv  Detail & Related papers  (2021-06-04T09:48:28Z)
• Infusing Finetuning with Semantic Dependencies [62.37697048781823]
  We show that, unlike syntax, semantics is not brought to the surface by today's pretrained models. We then use convolutional graph encoders to explicitly incorporate semantic parses into task-specific finetuning.
  arXiv  Detail & Related papers  (2020-12-10T01:27:24Z)
• A Linguistic Analysis of Visually Grounded Dialogues Based on Spatial Expressions [35.24301299033675]
  We propose a framework for investigating fine-grained language understanding in visually grounded dialogues. We focus on OneCommon Corpus citepudagawa 2019natural,udagawa 2020annotated, a simple yet challenging common grounding dataset. We analyze their linguistic structures based on textitspatial expressions and provide comprehensive and reliable annotation for 600 dialogues.
  arXiv  Detail & Related papers  (2020-10-07T02:50:38Z)
• On the Branching Bias of Syntax Extracted from Pre-trained Language Models [47.82102426290707]
  We propose quantitatively measuring the branching bias by comparing the performance gap on a language and its reversed language. We analyze the impacts of three factors on the branching bias, namely parsing algorithms, feature definitions, and language models.
  arXiv  Detail & Related papers  (2020-10-06T03:09:14Z)
• How to Probe Sentence Embeddings in Low-Resource Languages: On Structural Design Choices for Probing Task Evaluation [82.96358326053115]
  We investigate sensitivity of probing task results to structural design choices. We probe embeddings in a multilingual setup with design choices that lie in a'stable region', as we identify for English. We find that results on English do not transfer to other languages.
  arXiv  Detail & Related papers  (2020-06-16T12:37:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.