Detecting agreement in multi-party dialogue: evaluating speaker diarisation versus a procedural baseline to enhance user engagement

• URL: http://arxiv.org/abs/2311.03021v1
• Date: Mon, 6 Nov 2023 11:00:44 GMT
• Title: Detecting agreement in multi-party dialogue: evaluating speaker diarisation versus a procedural baseline to enhance user engagement
• Abstract summary: This study uses a cooperative quiz, where the conversational agent acts as quiz-show host, to determine whether diarisation or a frequency-and-proximity-based method is more accurate at determining agreement. Experimental results show that our procedural system was more engaging to players, and was more accurate at detecting agreement.
• Score: 33.93135348317864
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Conversational agents participating in multi-party interactions face significant challenges in dialogue state tracking, since the identity of the speaker adds significant contextual meaning. It is common to utilise diarisation models to identify the speaker. However, it is not clear if these are accurate enough to correctly identify specific conversational events such as agreement or disagreement during a real-time interaction. This study uses a cooperative quiz, where the conversational agent acts as quiz-show host, to determine whether diarisation or a frequency-and-proximity-based method is more accurate at determining agreement, and whether this translates to feelings of engagement from the players. Experimental results show that our procedural system was more engaging to players, and was more accurate at detecting agreement, reaching an average accuracy of 0.44 compared to 0.28 for the diarised system.

Related papers

• MP-Bench: Evaluating Voice Agents as a Multiparty Conversation Participant [138.90198190949545]
  We introduce MP-Bench, the first benchmark specifically designed to evaluate conversational speech systems as active participants within multi-party contexts.<n>MP-Bench assesses agent behavior along two primary dimensions: turn-taking awareness and response appropriateness.<n>We find that real-time voice agents stay at or below 22% on multiparty comprehension and remain near chance on multiparty turn-taking.
  arXiv  Detail & Related papers  (2026-09-11T17:11:55Z)
• Analysing Cross-Speaker Convergence in Face-to-Face Dialogue through the Lens of Automatically Detected Shared Linguistic Constructions [4.216085185442862]
  This study applies a methodology for automatically detecting shared lemmatised constructions to a referential communication corpus. Our analyses uncover the usage patterns of shared constructions in interaction and reveal that features such as their frequency and the amount of different constructions used for a referent are associated with the degree of object labelling convergence.
  arXiv  Detail & Related papers  (2024-05-14T12:34:25Z)
• CKERC : Joint Large Language Models with Commonsense Knowledge for Emotion Recognition in Conversation [0.0]
  Emotion recognition in conversation (ERC) is a task which predicts the emotion of an utterance in the context of a conversation. We propose a novel joint large language models with commonsense knowledge framework for emotion recognition in conversation, namely CKERC.
  arXiv  Detail & Related papers  (2024-03-12T02:37:11Z)
• An Analysis of User Behaviors for Objectively Evaluating Spoken Dialogue Systems [26.003947740875482]
  We investigate the relationship between user behaviors and subjective evaluation scores in social dialogue tasks. The results reveal that in dialogue tasks where user utterances are primary, like attentive listening and job interview, indicators like the number of utterances and words play a significant role in evaluation.
  arXiv  Detail & Related papers  (2024-01-10T01:02:26Z)
• Self-supervised Speaker Recognition Training Using Human-Machine Dialogues [22.262550043863445]
  We investigate how to pretrain speaker recognition models by leveraging dialogues between customers and smart-speaker devices. We propose an effective rejection mechanism that selectively learns from dialogues based on their acoustic homogeneity. Experiments demonstrate that the proposed method provides significant performance improvements, superior to earlier work.
  arXiv  Detail & Related papers  (2022-02-07T19:44:54Z)
• "How Robust r u?": Evaluating Task-Oriented Dialogue Systems on Spoken Conversations [87.95711406978157]
  This work presents a new benchmark on spoken task-oriented conversations. We study multi-domain dialogue state tracking and knowledge-grounded dialogue modeling. Our data set enables speech-based benchmarking of task-oriented dialogue systems.
  arXiv  Detail & Related papers  (2021-09-28T04:51:04Z)
• DialogueBERT: A Self-Supervised Learning based Dialogue Pre-training Encoder [19.51263716065853]
  We propose a novel contextual dialogue encoder (i.e. DialogueBERT) based on the popular pre-trained language model BERT. Five self-supervised learning pre-training tasks are devised for learning the particularity of dialouge utterances. DialogueBERT was pre-trained with 70 million dialogues in real scenario, and then fine-tuned in three different downstream dialogue understanding tasks.
  arXiv  Detail & Related papers  (2021-09-22T01:41:28Z)
• Dialogue History Matters! Personalized Response Selectionin Multi-turn Retrieval-based Chatbots [62.295373408415365]
  We propose a personalized hybrid matching network (PHMN) for context-response matching. Our contributions are two-fold: 1) our model extracts personalized wording behaviors from user-specific dialogue history as extra matching information. We evaluate our model on two large datasets with user identification, i.e., personalized dialogue Corpus Ubuntu (P- Ubuntu) and personalized Weibo dataset (P-Weibo)
  arXiv  Detail & Related papers  (2021-03-17T09:42:11Z)
• A Data-driven Approach to Estimate User Satisfaction in Multi-turn Dialogues [22.298425763949634]
  The evaluation of multi-turn dialogues remains challenging. assigning the same experience score to two tasks with different complexity levels is misleading. We develop a new method to estimate the turn-level satisfaction for dialogue, which is context-sensitive and has a long-term view.
  arXiv  Detail & Related papers  (2021-03-01T20:00:28Z)
• Rethinking Dialogue State Tracking with Reasoning [76.0991910623001]
  This paper proposes to track dialogue states gradually with reasoning over dialogue turns with the help of the back-end data. Empirical results demonstrate that our method significantly outperforms the state-of-the-art methods by 38.6% in terms of joint belief accuracy for MultiWOZ 2.1.
  arXiv  Detail & Related papers  (2020-05-27T02:05:33Z)
• Learning an Unreferenced Metric for Online Dialogue Evaluation [53.38078951628143]
  We propose an unreferenced automated evaluation metric that uses large pre-trained language models to extract latent representations of utterances. We show that our model achieves higher correlation with human annotations in an online setting, while not requiring true responses for comparison during inference.
  arXiv  Detail & Related papers  (2020-05-01T20:01:39Z)
• You Impress Me: Dialogue Generation via Mutual Persona Perception [62.89449096369027]
  The research in cognitive science suggests that understanding is an essential signal for a high-quality chit-chat conversation. Motivated by this, we propose P2 Bot, a transmitter-receiver based framework with the aim of explicitly modeling understanding.
  arXiv  Detail & Related papers  (2020-04-11T12:51:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.