Detecting Agreement in Multi-party Conversational AI

• URL: http://arxiv.org/abs/2311.03026v1
• Date: Mon, 6 Nov 2023 11:04:39 GMT
• Title: Detecting Agreement in Multi-party Conversational AI
• Abstract summary: We present our work on a multi-party conversational system, which invites two users to play a trivia quiz game. The system detects users' agreement or disagreement on a final answer and responds accordingly. Our annotated transcripts and the code for the proposed system have been released open-source on GitHub.
• Score: 2.755416211770089
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Today, conversational systems are expected to handle conversations in multi-party settings, especially within Socially Assistive Robots (SARs). However, practical usability remains difficult as there are additional challenges to overcome, such as speaker recognition, addressee recognition, and complex turn-taking. In this paper, we present our work on a multi-party conversational system, which invites two users to play a trivia quiz game. The system detects users' agreement or disagreement on a final answer and responds accordingly. Our evaluation includes both performance and user assessment results, with a focus on detecting user agreement. Our annotated transcripts and the code for the proposed system have been released open-source on GitHub.

Related papers

• MP-Bench: Evaluating Voice Agents as a Multiparty Conversation Participant [138.90198190949545]
  We introduce MP-Bench, the first benchmark specifically designed to evaluate conversational speech systems as active participants within multi-party contexts.<n>MP-Bench assesses agent behavior along two primary dimensions: turn-taking awareness and response appropriateness.<n>We find that real-time voice agents stay at or below 22% on multiparty comprehension and remain near chance on multiparty turn-taking.
  arXiv  Detail & Related papers  (2026-09-11T17:11:55Z)
• Say It My Way: Exploring Control in Conversational Visual Question Answering with Blind Users [8.54162000547885]
  We show how users can customize visual question answering (VQA) tools with a real-world conversational VQA system.<n>The system lacked verbosity controls, was limited in estimating distance in space and time, relied on inaccessible image framing, and offered little to no camera guidance.<n>We discuss how customization techniques such as prompt engineering can help participants work around these limitations.
  arXiv  Detail & Related papers  (2026-02-18T22:44:43Z)
• ESPnet-SDS: Unified Toolkit and Demo for Spoken Dialogue Systems [57.806797579986075]
  We introduce an open-source, user-friendly toolkit to build unified web interfaces for various cascaded and E2E spoken dialogue systems. Using the evaluation metrics, we compare various cascaded and E2E spoken dialogue systems with a human-human conversation dataset as a proxy. Our analysis demonstrates that the toolkit allows researchers to effortlessly compare and contrast different technologies.
  arXiv  Detail & Related papers  (2025-03-11T15:24:02Z)
• Rethinking the Evaluation of Dialogue Systems: Effects of User Feedback on Crowdworkers and LLMs [57.16442740983528]
  In ad-hoc retrieval, evaluation relies heavily on user actions, including implicit feedback. The role of user feedback in annotators' assessment of turns in a conversational perception has been little studied. We focus on how the evaluation of task-oriented dialogue systems ( TDSs) is affected by considering user feedback, explicit or implicit, as provided through the follow-up utterance of a turn being evaluated.
  arXiv  Detail & Related papers  (2024-04-19T16:45:50Z)
• Detecting agreement in multi-party dialogue: evaluating speaker diarisation versus a procedural baseline to enhance user engagement [33.93135348317864]
  This study uses a cooperative quiz, where the conversational agent acts as quiz-show host, to determine whether diarisation or a frequency-and-proximity-based method is more accurate at determining agreement. Experimental results show that our procedural system was more engaging to players, and was more accurate at detecting agreement.
  arXiv  Detail & Related papers  (2023-11-06T11:00:44Z)
• Multi-User MultiWOZ: Task-Oriented Dialogues among Multiple Users [51.34484827552774]
  We release the Multi-User MultiWOZ dataset: task-oriented dialogues among two users and one agent. These dialogues reflect interesting dynamics of collaborative decision-making in task-oriented scenarios. We propose a novel task of multi-user contextual query rewriting: to rewrite a task-oriented chat between two users as a concise task-oriented query.
  arXiv  Detail & Related papers  (2023-10-31T14:12:07Z)
• PICK: Polished & Informed Candidate Scoring for Knowledge-Grounded Dialogue Systems [59.1250765143521]
  Current knowledge-grounded dialogue systems often fail to align the generated responses with human-preferred qualities. We propose Polished & Informed Candidate Scoring (PICK), a generation re-scoring framework. We demonstrate the effectiveness of PICK in generating responses that are more faithful while keeping them relevant to the dialogue history.
  arXiv  Detail & Related papers  (2023-09-19T08:27:09Z)
• Multi-Action Dialog Policy Learning from Logged User Feedback [28.4271696269512]
  Multi-action dialog policy generates multiple atomic dialog actions per turn. Due to data limitations, existing policy models generalize poorly toward unseen dialog flows. We propose BanditMatch to improve multi-action dialog policy learning with explicit and implicit turn-level user feedback.
  arXiv  Detail & Related papers  (2023-02-27T04:01:28Z)
• Are Akpans Trick or Treat: Unveiling Helpful Biases in Assistant Systems [55.09907990139756]
  Information-seeking AI assistant systems aim to answer users' queries about knowledge in a timely manner. In this paper, we study computational measurements of helpfulness. Experiments with state-of-the-art dialogue systems reveal that existing systems tend to be more helpful for questions regarding concepts from highly-developed countries.
  arXiv  Detail & Related papers  (2022-05-25T07:58:38Z)
• Multi-stage Clarification in Conversational AI: The case of Question-Answering Dialogue Systems [0.27998963147546135]
  Clarification resolution plays an important role in various information retrieval tasks such as interactive question answering and conversational search. We propose a multi-stage clarification mechanism for prompting clarification and query selection in the context of a question answering dialogue system. Our proposed mechanism improves the overall user experience and outperforms competitive baselines with two datasets.
  arXiv  Detail & Related papers  (2021-10-28T15:45:44Z)
• Hierarchical Summarization for Longform Spoken Dialog [1.995792341399967]
  Despite the pervasiveness of spoken dialog, automated speech understanding and quality information extraction remains markedly poor. Compared to understanding text, auditory communication poses many additional challenges such as speaker disfluencies, informal prose styles, and lack of structure. We propose a two stage ASR and text summarization pipeline and propose a set of semantic segmentation and merging algorithms to resolve these speech modeling challenges.
  arXiv  Detail & Related papers  (2021-08-21T23:31:31Z)
• Improving Conversational Question Answering Systems after Deployment using Feedback-Weighted Learning [69.42679922160684]
  We propose feedback-weighted learning based on importance sampling to improve upon an initial supervised system using binary user feedback. Our work opens the prospect to exploit interactions with real users and improve conversational systems after deployment.
  arXiv  Detail & Related papers  (2020-11-01T19:50:34Z)
• Towards Data Distillation for End-to-end Spoken Conversational Question Answering [65.124088336738]
  We propose a new Spoken Conversational Question Answering task (SCQA) SCQA aims at enabling QA systems to model complex dialogues flow given the speech utterances and text corpora. Our main objective is to build a QA system to deal with conversational questions both in spoken and text forms.
  arXiv  Detail & Related papers  (2020-10-18T05:53:39Z)
• ConvAI3: Generating Clarifying Questions for Open-Domain Dialogue Systems (ClariQ) [64.60303062063663]
  This document presents a detailed description of the challenge on clarifying questions for dialogue systems (ClariQ) The challenge is organized as part of the Conversational AI challenge series (ConvAI3) at Search Oriented Conversational AI (SCAI) EMNLP workshop in 2020.
  arXiv  Detail & Related papers  (2020-09-23T19:48:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.