WMT24 Test Suite: Gender Resolution in Speaker-Listener Dialogue Roles

• URL: http://arxiv.org/abs/2411.06194v1
• Date: Sat, 09 Nov 2024 14:30:58 GMT
• Title: WMT24 Test Suite: Gender Resolution in Speaker-Listener Dialogue Roles
• Abstract summary: We assess the difficulty of gender resolution in literary-style dialogue settings. We find that character and manner stereotypes outside of the dialogue significantly impact the gender agreement of referents within the dialogue.
• Score: 13.511723323294339
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We assess the difficulty of gender resolution in literary-style dialogue settings and the influence of gender stereotypes. Instances of the test suite contain spoken dialogue interleaved with external meta-context about the characters and the manner of speaking. We find that character and manner stereotypes outside of the dialogue significantly impact the gender agreement of referents within the dialogue.

Related papers

• You Talkin to Me?: A Network Analysis of Gendered Speaker-Addressee Patterns in Film Screenplays [58.97068363450271]
  We use a dataset of 4,600 directed dialogue events from 38 film screenplays.<n>Male characters dominate as both speakers and addressees corpus wide, even in scenes with more women.<n>Cross gender dialogue is directionally symmetric on average but clustered at the film level.
  arXiv  Detail & Related papers  (2026-06-26T17:38:19Z)
• Context Does Matter: Implications for Crowdsourced Evaluation Labels in Task-Oriented Dialogue Systems [57.16442740983528]
  Crowdsourced labels play a crucial role in evaluating task-oriented dialogue systems. Previous studies suggest using only a portion of the dialogue context in the annotation process. This study investigates the influence of dialogue context on annotation quality.
  arXiv  Detail & Related papers  (2024-04-15T17:56:39Z)
• Different Games in Dialogue: Combining character and conversational types in strategic choice [0.0]
  We show that investigating the interaction of conversational type with the character types of the interlocutors is worthwhile. We present a method of calculating the decision making process for selecting dialogue moves that combines character type and conversational type.
  arXiv  Detail & Related papers  (2023-07-05T07:51:47Z)
• A Benchmark for Understanding and Generating Dialogue between Characters in Stories [75.29466820496913]
  We present the first study to explore whether machines can understand and generate dialogue in stories. We propose two new tasks including Masked Dialogue Generation and Dialogue Speaker Recognition. We show the difficulty of the proposed tasks by testing existing models with automatic and manual evaluation on DialStory.
  arXiv  Detail & Related papers  (2022-09-18T10:19:04Z)
• Structural Modeling for Dialogue Disentanglement [43.352833140317486]
  Multi-party dialogue context Tangled multi-party dialogue context leads to challenges for dialogue reading comprehension. This work designs a novel model to disentangle multi-party history into threads, by taking dialogue structure features into account.
  arXiv  Detail & Related papers  (2021-10-15T11:28:43Z)
• "How Robust r u?": Evaluating Task-Oriented Dialogue Systems on Spoken Conversations [87.95711406978157]
  This work presents a new benchmark on spoken task-oriented conversations. We study multi-domain dialogue state tracking and knowledge-grounded dialogue modeling. Our data set enables speech-based benchmarking of task-oriented dialogue systems.
  arXiv  Detail & Related papers  (2021-09-28T04:51:04Z)
• Mitigating Gender Bias for Neural Dialogue Generation with Adversarial Learning [44.69720475052093]
  We propose a novel adversarial learning framework Debiased-Chat to train dialogue models free from gender bias. Our framework significantly reduces gender bias in dialogue models while maintaining the response quality.
  arXiv  Detail & Related papers  (2020-09-28T02:46:59Z)
• Is this Dialogue Coherent? Learning from Dialogue Acts and Entities [82.44143808977209]
  We create the Switchboard Coherence (SWBD-Coh) corpus, a dataset of human-human spoken dialogues annotated with turn coherence ratings. Our statistical analysis of the corpus indicates how turn coherence perception is affected by patterns of distribution of entities. We find that models combining both DA and entity information yield the best performances both for response selection and turn coherence rating.
  arXiv  Detail & Related papers  (2020-06-17T21:02:40Z)
• Rethinking Dialogue State Tracking with Reasoning [76.0991910623001]
  This paper proposes to track dialogue states gradually with reasoning over dialogue turns with the help of the back-end data. Empirical results demonstrate that our method significantly outperforms the state-of-the-art methods by 38.6% in terms of joint belief accuracy for MultiWOZ 2.1.
  arXiv  Detail & Related papers  (2020-05-27T02:05:33Z)
• Dialogue-Based Relation Extraction [53.2896545819799]
  We present the first human-annotated dialogue-based relation extraction (RE) dataset DialogRE. We argue that speaker-related information plays a critical role in the proposed task, based on an analysis of similarities and differences between dialogue-based and traditional RE tasks. Experimental results demonstrate that a speaker-aware extension on the best-performing model leads to gains in both the standard and conversational evaluation settings.
  arXiv  Detail & Related papers  (2020-04-17T03:51:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.