On Crowdsourcing Task Design for Discourse Relation Annotation

• URL: http://arxiv.org/abs/2412.11637v1
• Date: Mon, 16 Dec 2024 10:26:11 GMT
• Title: On Crowdsourcing Task Design for Discourse Relation Annotation
• Authors: Frances Yung, Vera Demberg,
• Abstract summary: We compare two methods that crowdsource English implicit discourse relation annotation by connective insertion. We re-annotate the whole DiscoGeM 1.0 corpus using the forced-choice approach. Comparison among over 130,000 annotations, however, shows that the free-choice strategy produces less diverse annotations.
• Score: 13.996171129586731
• License:
• Abstract: Interpreting implicit discourse relations involves complex reasoning, requiring the integration of semantic cues with background knowledge, as overt connectives like because or then are absent. These relations often allow multiple interpretations, best represented as distributions. In this study, we compare two established methods that crowdsource English implicit discourse relation annotation by connective insertion: a free-choice approach, which allows annotators to select any suitable connective, and a forced-choice approach, which asks them to select among a set of predefined options. Specifically, we re-annotate the whole DiscoGeM 1.0 corpus -- initially annotated with the free-choice method -- using the forced-choice approach. The free-choice approach allows for flexible and intuitive insertion of various connectives, which are context-dependent. Comparison among over 130,000 annotations, however, shows that the free-choice strategy produces less diverse annotations, often converging on common labels. Analysis of the results reveals the interplay between task design and the annotators' abilities to interpret and produce discourse relations.

Related papers

• Tuning-Free Personalized Alignment via Trial-Error-Explain In-Context Learning [74.56097953187994]
  We present Trial-Error-Explain In-Context Learning (TICL), a tuning-free method that personalizes language models for text generation tasks. TICL iteratively expands an in-context learning prompt via a trial-error-explain process, adding model-generated negative samples and explanations. TICL achieves up to 91.5% against the previous state-of-the-art and outperforms competitive tuning-free baselines for personalized alignment tasks.
  arXiv  Detail & Related papers  (2025-02-13T05:20:21Z)
• Leveraging Hierarchical Prototypes as the Verbalizer for Implicit Discourse Relation Recognition [7.149359970799236]
  Implicit discourse relation recognition involves determining relationships between spans of text that are not linked by an explicit discourse connective. Previous work solely relied on manual verbalizers for implicit discourse relation recognition. We leverage the prototypes that capture certain class-level semantic features and the hierarchical label structure for different classes as the verbalizer.
  arXiv  Detail & Related papers  (2024-11-22T12:01:04Z)
• DenoSent: A Denoising Objective for Self-Supervised Sentence Representation Learning [59.4644086610381]
  We propose a novel denoising objective that inherits from another perspective, i.e., the intra-sentence perspective. By introducing both discrete and continuous noise, we generate noisy sentences and then train our model to restore them to their original form. Our empirical evaluations demonstrate that this approach delivers competitive results on both semantic textual similarity (STS) and a wide range of transfer tasks.
  arXiv  Detail & Related papers  (2024-01-24T17:48:45Z)
• IDEAL: Influence-Driven Selective Annotations Empower In-Context Learners in Large Language Models [66.32043210237768]
  This paper introduces an influence-driven selective annotation method. It aims to minimize annotation costs while improving the quality of in-context examples. Experiments confirm the superiority of the proposed method on various benchmarks.
  arXiv  Detail & Related papers  (2023-10-16T22:53:54Z)
• Design Choices for Crowdsourcing Implicit Discourse Relations: Revealing the Biases Introduced by Task Design [23.632204469647526]
  We show that the task design can push annotators towards certain relations. We conclude that this type of bias should be taken into account when training and testing models.
  arXiv  Detail & Related papers  (2023-04-03T09:04:18Z)
• Improve Retrieval-based Dialogue System via Syntax-Informed Attention [46.79601705850277]
  We propose SIA, Syntax-Informed Attention, considering both intra- and inter-sentence syntax information. We evaluate our method on three widely used benchmarks and experimental results demonstrate the general superiority of our method on dialogue response selection.
  arXiv  Detail & Related papers  (2023-03-12T08:14:16Z)
• Relational Sentence Embedding for Flexible Semantic Matching [86.21393054423355]
  We present Sentence Embedding (RSE), a new paradigm to discover further the potential of sentence embeddings. RSE is effective and flexible in modeling sentence relations and outperforms a series of state-of-the-art embedding methods.
  arXiv  Detail & Related papers  (2022-12-17T05:25:17Z)
• Natural Language Inference Prompts for Zero-shot Emotion Classification in Text across Corpora [11.986676998327864]
  We show that the choice of a particular prompt formulation needs to fit to the corpus. We show that this challenge can be tackled with combinations of multiple prompts.
  arXiv  Detail & Related papers  (2022-09-14T15:07:36Z)
• Relation-aware Video Reading Comprehension for Temporal Language Grounding [67.5613853693704]
  Temporal language grounding in videos aims to localize the temporal span relevant to the given query sentence. This paper will formulate temporal language grounding into video reading comprehension and propose a Relation-aware Network (RaNet) to address it.
  arXiv  Detail & Related papers  (2021-10-12T03:10:21Z)
• Learning to Decouple Relations: Few-Shot Relation Classification with Entity-Guided Attention and Confusion-Aware Training [49.9995628166064]
  We propose CTEG, a model equipped with two mechanisms to learn to decouple easily-confused relations. On the one hand, an EGA mechanism is introduced to guide the attention to filter out information causing confusion. On the other hand, a Confusion-Aware Training (CAT) method is proposed to explicitly learn to distinguish relations.
  arXiv  Detail & Related papers  (2020-10-21T11:07:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.