MP-Bench: Evaluating Voice Agents as a Multiparty Conversation Participant

• URL: http://arxiv.org/abs/2609.13076v1
• Date: Fri, 11 Sep 2026 17:11:55 GMT
• Title: MP-Bench: Evaluating Voice Agents as a Multiparty Conversation Participant
• Abstract summary: We introduce MP-Bench, the first benchmark specifically designed to evaluate conversational speech systems as active participants within multi-party contexts.<n>MP-Bench assesses agent behavior along two primary dimensions: turn-taking awareness and response appropriateness.<n>We find that real-time voice agents stay at or below 22% on multiparty comprehension and remain near chance on multiparty turn-taking.
• Score: 138.90198190949545
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Conversational voice agents have advanced significantly, offering increasingly natural human-machine interactions through both cascaded and end-to-end architectures. However, while recent benchmarks extensively evaluate dyadic interactions and passive audio comprehension, they largely overlook a prevalent real-world scenario: multi-party conversations. Evaluating agents in these settings is fundamentally more challenging than in dyadic interactions due to the exponentially greater conversational complexity. For voice agents to integrate seamlessly into human group dynamics, they must not only generate contextually appropriate responses but also demonstrate a nuanced understanding of open turn-taking. To address this gap, we introduce Multiparty Bench (MP-Bench), the first benchmark specifically designed to objectively evaluate conversational speech systems as active participants within multi-party contexts. MP-Bench assesses agent behavior along two primary dimensions: turn-taking awareness and response appropriateness. Additionally, we incorporate comprehension-based question-answering tasks as a complementary evaluation. By benchmarking 12 voice agents, we find that real-time voice agents stay at or below 22% on multiparty comprehension and remain near chance on multiparty turn-taking, exposing an open challenge for real-time voice agents under multiparty scenario.

Related papers

• Who Are They to Each Other? Multi-Agent Reasoning for Speaker Relationship Inference [18.334093251358833]
  Inferring speaker relationships from spoken conversations is an important step towards socially aware speech understanding.<n>Existing inference-time LLM approaches provide limited structure for handling subtle, distributed, and multimodal relational cues.<n>We introduce a training-free multi-agent reasoning framework that organizes inference through structured interaction among LLM agents.
  arXiv  Detail & Related papers  (2026-09-09T02:40:02Z)
• Adaptive Turn-Taking for Real-time Multi-Party Voice Agents [4.515705397557082]
  We propose ModeratorLM, a role-playing voice agent that conditions turn-taking behavior on an explicitly assigned role in multi-party settings.<n>We construct RolePlayConv, a large-scale synthetic dataset of spoken multi-party conversations with diverse assistant roles.
  arXiv  Detail & Related papers  (2026-06-11T16:27:45Z)
• Covo-Audio Technical Report [61.09708870154148]
  Covo-Audio, a 7B-end LALM, directly processes continuous audio inputs and generates audio outputs within a single unified architecture.<n>Covo-Audio-Chat, a dialogue-oriented variant, demonstrates semantic strong spoken conversational abilities.
  arXiv  Detail & Related papers  (2026-02-10T14:31:11Z)
• Chronological Thinking in Full-Duplex Spoken Dialogue Language Models [66.84843878538207]
  Chronological Thinking aims to improve response quality in full SDLMs.<n>No additional latency: once the user stops speaking, the agent halts thinking and begins speaking without further delay.<n>Results: Experiments demonstrate the effectiveness of chronological thinking through both objective metrics and human evaluations.
  arXiv  Detail & Related papers  (2025-10-02T10:28:11Z)
• SpeechRole: A Large-Scale Dataset and Benchmark for Evaluating Speech Role-Playing Agents [72.79816494079833]
  Role-playing agents have emerged as a promising paradigm for achieving personalized interaction and emotional resonance.<n>Existing research primarily focuses on the textual modality, neglecting the critical dimension of speech in realistic interactive scenarios.<n>We construct SpeechRole-Data, a large-scale, high-quality dataset that comprises 98 diverse roles and 112k speech-based single-turn and multi-turn conversations.
  arXiv  Detail & Related papers  (2025-08-04T03:18:36Z)
• Multimodal Conversation Structure Understanding [12.29827265137757]
  Large language models' ability to understand fine-grained conversational structure remains underexplored.<n>We present a human annotated dataset of 4,398 annotations for speakers and reply-to relationship, 5,755 addressees, and 3,142 side-participants.<n>We evaluate popular audio-visual LLMs and vision-language models on our dataset, and our experimental results suggest that multimodal conversational structure understanding remains challenging.
  arXiv  Detail & Related papers  (2025-05-23T06:41:54Z)
• Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities [93.09944267871163]
  FullDuplexBench is a benchmark that systematically evaluates key interactive behaviors.<n>By releasing our benchmark code we aim to advance spoken dialogue modeling and the development of more natural and engaging SDMs.
  arXiv  Detail & Related papers  (2025-03-06T18:59:16Z)
• Who Speaks Next? Multi-party AI Discussion Leveraging the Systematics of Turn-taking in Murder Mystery Games [7.946510318969307]
  This study focuses on conversational norms such as adjacency pairs and turn-taking found in conversation analysis.<n>We propose a new framework called "Murder Mystery Agents" that applies these norms to AI agents' dialogue control.<n>The proposed framework integrates next speaker selection based on adjacency pairs and a self-selection mechanism.
  arXiv  Detail & Related papers  (2024-12-06T10:45:54Z)
• Detecting agreement in multi-party dialogue: evaluating speaker diarisation versus a procedural baseline to enhance user engagement [33.93135348317864]
  This study uses a cooperative quiz, where the conversational agent acts as quiz-show host, to determine whether diarisation or a frequency-and-proximity-based method is more accurate at determining agreement. Experimental results show that our procedural system was more engaging to players, and was more accurate at detecting agreement.
  arXiv  Detail & Related papers  (2023-11-06T11:00:44Z)
• Duplex Conversation: Towards Human-like Interaction in Spoken Dialogue System [120.70726465994781]
  multimodal spoken dialogue system enables telephonebased agents to interact with customers like human. We deploy Conversation Duplex Alibaba intelligent customer service to share lessons learned in production. Online A/B experiments show in proposed system can significantly reduce response latency by 50%.
  arXiv  Detail & Related papers  (2022-05-30T12:41:23Z)
• A Speaker-aware Parallel Hierarchical Attentive Encoder-Decoder Model for Multi-turn Dialogue Generation [13.820298189734686]
  This paper presents a novel open-domain dialogue generation model emphasizing the differentiation of speakers in multi-turn conversations. Our empirical results show that PHAED outperforms the state-of-the-art in both automatic and human evaluations.
  arXiv  Detail & Related papers  (2021-10-13T16:08:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.