Incorporating Annotator Uncertainty into Representations of Discourse Relations

• URL: http://arxiv.org/abs/2308.07179v1
• Date: Mon, 14 Aug 2023 14:39:02 GMT
• Title: Incorporating Annotator Uncertainty into Representations of Discourse Relations
• Authors: S. Magal\'i L\'opez Cortez and Cassandra L. Jacobs
• Abstract summary: We find that dialogue context is a significant predictor of confidence scores. We show that weighting discourse relation representations with information about confidence and dialogue context coherently models our annotators' uncertainty about discourse relation labels.
• Score: 12.110722011252681
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Annotation of discourse relations is a known difficult task, especially for non-expert annotators. In this paper, we investigate novice annotators' uncertainty on the annotation of discourse relations on spoken conversational data. We find that dialogue context (single turn, pair of turns within speaker, and pair of turns across speakers) is a significant predictor of confidence scores. We compute distributed representations of discourse relations from co-occurrence statistics that incorporate information about confidence scores and dialogue context. We perform a hierarchical clustering analysis using these representations and show that weighting discourse relation representations with information about confidence and dialogue context coherently models our annotators' uncertainty about discourse relation labels.

Related papers

• The distribution of discourse relations within and across turns in spontaneous conversation [13.053424646592749]
  Time pressure and topic negotiation may impose constraints on how people leverage discourse relations. We adapt a system of DRs for written language to spontaneous dialogue using crowdsourced annotations from novice annotators.
  arXiv  Detail & Related papers  (2023-07-07T15:06:31Z)
• Pre-training Multi-party Dialogue Models with Latent Discourse Inference [85.9683181507206]
  We pre-train a model that understands the discourse structure of multi-party dialogues, namely, to whom each utterance is replying. To fully utilize the unlabeled data, we propose to treat the discourse structures as latent variables, then jointly infer them and pre-train the discourse-aware model.
  arXiv  Detail & Related papers  (2023-05-24T14:06:27Z)
• ChatGPT Evaluation on Sentence Level Relations: A Focus on Temporal, Causal, and Discourse Relations [52.26802326949116]
  We quantitatively evaluate the performance of ChatGPT, an interactive large language model, on inter-sentential relations. ChatGPT exhibits exceptional proficiency in detecting and reasoning about causal relations. It is capable of identifying the majority of discourse relations with existing explicit discourse connectives, but the implicit discourse relation remains a formidable challenge.
  arXiv  Detail & Related papers  (2023-04-28T13:14:36Z)
• "How Robust r u?": Evaluating Task-Oriented Dialogue Systems on Spoken Conversations [87.95711406978157]
  This work presents a new benchmark on spoken task-oriented conversations. We study multi-domain dialogue state tracking and knowledge-grounded dialogue modeling. Our data set enables speech-based benchmarking of task-oriented dialogue systems.
  arXiv  Detail & Related papers  (2021-09-28T04:51:04Z)
• Speaker-Oriented Latent Structures for Dialogue-Based Relation Extraction [10.381257436462116]
  We introduce SOLS, a novel model which can explicitly induce speaker-oriented latent structures for better DiaRE. Specifically, we learn latent structures to capture the relationships among tokens beyond the utterance boundaries. During the learning process, our speaker-specific regularization method progressively highlights speaker-related key clues and erases the irrelevant ones.
  arXiv  Detail & Related papers  (2021-09-11T04:24:51Z)
• Enhanced Speaker-aware Multi-party Multi-turn Dialogue Comprehension [43.352833140317486]
  Multi-party multi-turn dialogue comprehension brings unprecedented challenges. Most existing methods deal with dialogue contexts as plain texts. We propose an enhanced speaker-aware model with masking attention and heterogeneous graph networks.
  arXiv  Detail & Related papers  (2021-09-09T07:12:22Z)
• Self- and Pseudo-self-supervised Prediction of Speaker and Key-utterance for Multi-party Dialogue Reading Comprehension [46.69961067676279]
  Multi-party dialogue machine reading comprehension (MRC) brings tremendous challenge since it involves multiple speakers at one dialogue. Previous models focus on how to incorporate speaker information flows using complex graph-based modules. In this paper, we design two labour-free self- and pseudo-self-supervised prediction tasks on speaker and key-utterance to implicitly model the speaker information flows.
  arXiv  Detail & Related papers  (2021-09-08T16:51:41Z)
• I like fish, especially dolphins: Addressing Contradictions in Dialogue Modeling [104.09033240889106]
  We introduce the DialoguE COntradiction DEtection task (DECODE) and a new conversational dataset containing both human-human and human-bot contradictory dialogues. We then compare a structured utterance-based approach of using pre-trained Transformer models for contradiction detection with the typical unstructured approach.
  arXiv  Detail & Related papers  (2020-12-24T18:47:49Z)
• Is this Dialogue Coherent? Learning from Dialogue Acts and Entities [82.44143808977209]
  We create the Switchboard Coherence (SWBD-Coh) corpus, a dataset of human-human spoken dialogues annotated with turn coherence ratings. Our statistical analysis of the corpus indicates how turn coherence perception is affected by patterns of distribution of entities. We find that models combining both DA and entity information yield the best performances both for response selection and turn coherence rating.
  arXiv  Detail & Related papers  (2020-06-17T21:02:40Z)
• Dialogue-Based Relation Extraction [53.2896545819799]
  We present the first human-annotated dialogue-based relation extraction (RE) dataset DialogRE. We argue that speaker-related information plays a critical role in the proposed task, based on an analysis of similarities and differences between dialogue-based and traditional RE tasks. Experimental results demonstrate that a speaker-aware extension on the best-performing model leads to gains in both the standard and conversational evaluation settings.
  arXiv  Detail & Related papers  (2020-04-17T03:51:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.