Adaptive Turn-Taking for Real-time Multi-Party Voice Agents
- URL: http://arxiv.org/abs/2606.13544v1
- Date: Thu, 11 Jun 2026 16:27:45 GMT
- Title: Adaptive Turn-Taking for Real-time Multi-Party Voice Agents
- Abstract summary: We propose ModeratorLM, a role-playing voice agent that conditions turn-taking behavior on an explicitly assigned role in multi-party settings.<n>We construct RolePlayConv, a large-scale synthetic dataset of spoken multi-party conversations with diverse assistant roles.
- Score: 4.515705397557082
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: Turn-taking in multi-party spoken conversations remains a fundamental challenge for voice-based agents, particularly under dynamic floor competition and varying user expectations. We propose ModeratorLM, a role-playing voice agent that conditions turn-taking behavior on an explicitly assigned role in multi-party settings. The system is built on a speech large language model operating in chunk-wise streaming manner. We further introduce a reasoning-augmented variant that incorporates chain-of-thought reasoning over conversational context and the assigned role. We construct RolePlayConv, a large-scale synthetic dataset of spoken multi-party conversations with diverse assistant roles. Experiments on real-world meeting data and RolePlayConv show improved turn-taking precision by over 40% and recall by more than 70%, while substantially reducing false-positive interruptions compared to non-role-conditioned baselines.
Related papers
- MP-Bench: Evaluating Voice Agents as a Multiparty Conversation Participant [138.90198190949545]
We introduce MP-Bench, the first benchmark specifically designed to evaluate conversational speech systems as active participants within multi-party contexts.<n>MP-Bench assesses agent behavior along two primary dimensions: turn-taking awareness and response appropriateness.<n>We find that real-time voice agents stay at or below 22% on multiparty comprehension and remain near chance on multiparty turn-taking.
arXiv Detail & Related papers (2026-09-11T17:11:55Z) - Character Beyond Speech: Leveraging Role-Playing Evaluation in Audio Large Language Models via Reinforcement Learning [57.22705949022221]
RoleJudge is an evaluation framework that leverages audio large language models to assess the alignment between speech and character.<n>We introduce RoleChat, the first voice role-playing evaluation dataset enriched with chain-of-thought reasoning annotations.
arXiv Detail & Related papers (2026-04-15T12:39:03Z) - AdaMARP: An Adaptive Multi-Agent Interaction Framework for General Immersive Role-Playing [71.66362858228418]
LLM role-playing aims to portray arbitrary characters in interactive narratives, yet existing systems often suffer from limited immersion and adaptability.<n>We propose an adaptive multi-agent role-playing framework, AdaMARP, featuring an immersive message format that interleaves [Thought], (Action), Environment>, and Speech.
arXiv Detail & Related papers (2026-01-16T05:41:45Z) - PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models [33.33273575953341]
We introduce PersonaPlex, a duplex conversational speech model that incorporates hybrid system prompts.<n> PersonaPlex is trained on a large-scale synthetic dataset of paired prompts and user-agent conversations.<n>Experiments show that PersonaPlex achieves strong role-conditioned behavior, voice-conditioned speech, and natural conversational responsiveness.
arXiv Detail & Related papers (2026-01-14T07:47:46Z) - SpeechRole: A Large-Scale Dataset and Benchmark for Evaluating Speech Role-Playing Agents [72.79816494079833]
Role-playing agents have emerged as a promising paradigm for achieving personalized interaction and emotional resonance.<n>Existing research primarily focuses on the textual modality, neglecting the critical dimension of speech in realistic interactive scenarios.<n>We construct SpeechRole-Data, a large-scale, high-quality dataset that comprises 98 diverse roles and 112k speech-based single-turn and multi-turn conversations.
arXiv Detail & Related papers (2025-08-04T03:18:36Z) - OmniCharacter: Towards Immersive Role-Playing Agents with Seamless Speech-Language Personality Interaction [123.89581506075461]
We propose OmniCharacter, a first seamless speech-language personality interaction model to achieve immersive RPAs with low latency.<n> Specifically, OmniCharacter enables agents to consistently exhibit role-specific personality traits and vocal traits throughout the interaction.<n>Our method yields better responses in terms of both content and style compared to existing RPAs and mainstream speech-language models, with a response latency as low as 289ms.
arXiv Detail & Related papers (2025-05-26T17:55:06Z) - Multimodal Conversation Structure Understanding [12.29827265137757]
Large language models' ability to understand fine-grained conversational structure remains underexplored.<n>We present a human annotated dataset of 4,398 annotations for speakers and reply-to relationship, 5,755 addressees, and 3,142 side-participants.<n>We evaluate popular audio-visual LLMs and vision-language models on our dataset, and our experimental results suggest that multimodal conversational structure understanding remains challenging.
arXiv Detail & Related papers (2025-05-23T06:41:54Z) - Filling the Gap of Utterance-aware and Speaker-aware Representation for
Multi-turn Dialogue [76.88174667929665]
A multi-turn dialogue is composed of multiple utterances from two or more different speaker roles.
In the existing retrieval-based multi-turn dialogue modeling, the pre-trained language models (PrLMs) as encoder represent the dialogues coarsely.
We propose a novel model to fill such a gap by modeling the effective utterance-aware and speaker-aware representations entailed in a dialogue history.
arXiv Detail & Related papers (2020-09-14T15:07:19Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.