RoleBreak: Benchmarking Long-Horizon Role-Playing Robustness in Spoken Dialogue

• URL: http://arxiv.org/abs/2609.16614v1
• Date: Tue, 15 Sep 2026 04:19:53 GMT
• Title: RoleBreak: Benchmarking Long-Horizon Role-Playing Robustness in Spoken Dialogue
• Abstract summary: We introduce RoleBreak, an open benchmark for long-horizon role-playing robustness in spoken dialogue.<n>RoleBreak contains 310 character-based and user-centered roles, 6,688 human-verified dialogue turns, and 11,743 finegrained evaluation criteria.<n>We find four key patterns. First, current systems are substantially stronger at semantic role adherence than at vocal emotion.<n>Second, semantic robustness remains brittle over long interactions: even strongest evaluated system encounters its first persona and safety failures after only 10.4 and 11.6 turns on average.
• Score: 14.12516754496197
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Speech-to-speech dialogue models increasingly support persona control, yet existing spoken role-playing benchmarks remain largely character-centric and short-horizon. This leaves open whether spoken dialogue models can sustain diverse roles over extended interactions, especially beyond predefined fictional characters. We introduce RoleBreak, an open benchmark for long-horizon role-playing robustness in spoken dialogue. RoleBreak contains 310 character-based and user-centered roles, 6,688 human-verified dialogue turns, and 11,743 fine-grained evaluation criteria, with 1,856 turns carrying expressive emotion targets for evaluating vocal emotion. Its scenarios are designed to stress role consistency, interaction quality, safety, and affect over extended conversations. We evaluate nine configurations spanning full-duplex, omni-modal, and cascaded ASR--LLM--TTS paradigms. We find four key patterns. First, current systems are substantially stronger at semantic role adherence than at vocal emotion. Second, semantic robustness remains brittle over long interactions: even the strongest evaluated system encounters its first persona and safety failures after only 10.4 and 11.6 turns on average. Third, scaling the LLM substantially improves semantic robustness and delays failure, but yields little improvement in vocal emotion. Finally, user vocal emotion affects role-playing behavior even when linguistic content is fixed. These findings highlight persistent gaps in both long-horizon robustness and vocal expressiveness in spoken role-playing systems.

Related papers

• Adaptive Turn-Taking for Real-time Multi-Party Voice Agents [4.515705397557082]
  We propose ModeratorLM, a role-playing voice agent that conditions turn-taking behavior on an explicitly assigned role in multi-party settings.<n>We construct RolePlayConv, a large-scale synthetic dataset of spoken multi-party conversations with diverse assistant roles.
  arXiv  Detail & Related papers  (2026-06-11T16:27:45Z)
• Character Beyond Speech: Leveraging Role-Playing Evaluation in Audio Large Language Models via Reinforcement Learning [57.22705949022221]
  RoleJudge is an evaluation framework that leverages audio large language models to assess the alignment between speech and character.<n>We introduce RoleChat, the first voice role-playing evaluation dataset enriched with chain-of-thought reasoning annotations.
  arXiv  Detail & Related papers  (2026-04-15T12:39:03Z)
• ActorMind: Emulating Human Actor Reasoning for Speech Role-Playing [33.79520698647648]
  Speech Role-Playing enables models to deliver spontaneous responses with personalized verbal traits based on their role, the scene, and spoken dialogue.<n>ActorMind is an off-the-shelf, multi-agent, chain-of-though style reasoning framework that emulates how human actors perform in theaters.
  arXiv  Detail & Related papers  (2026-04-13T07:20:20Z)
• EchoMind: An Interrelated Multi-level Benchmark for Evaluating Empathetic Speech Language Models [47.41816926003011]
  Speech Language Models (SLMs) have made significant progress in spoken language understanding.<n>It remains unclear whether SLMs can fully perceive non lexical vocal cues alongside spoken words, and respond with empathy that aligns with both emotional and contextual factors.<n>We present EchoMind, the first interrelated, multi-level benchmark that simulates the cognitive process of empathetic dialogue.
  arXiv  Detail & Related papers  (2025-10-26T17:15:56Z)
• VoxRole: A Comprehensive Benchmark for Evaluating Speech-Based Role-Playing Agents [25.2769510663867]
  We introduce VoxRole, the first comprehensive benchmark specifically designed for the evaluation of speech-based RPCAs.<n>The benchmark comprises 13335 multi-turn dialogues, totaling 65.6 hours of speech from 1228 unique characters across 261 movies.
  arXiv  Detail & Related papers  (2025-09-04T07:03:46Z)
• SpeechRole: A Large-Scale Dataset and Benchmark for Evaluating Speech Role-Playing Agents [72.79816494079833]
  Role-playing agents have emerged as a promising paradigm for achieving personalized interaction and emotional resonance.<n>Existing research primarily focuses on the textual modality, neglecting the critical dimension of speech in realistic interactive scenarios.<n>We construct SpeechRole-Data, a large-scale, high-quality dataset that comprises 98 diverse roles and 112k speech-based single-turn and multi-turn conversations.
  arXiv  Detail & Related papers  (2025-08-04T03:18:36Z)
• HAMLET: Hyperadaptive Agent-based Modeling for Live Embodied Theatrics [48.154666939669205]
  HAMLET is a multi-agent framework focused on drama creation and online performance.<n>During the online performance, each actor is given an autonomous mind.<n>HamLET can create expressive and coherent theatrical experiences.
  arXiv  Detail & Related papers  (2025-07-21T11:36:39Z)
• OmniCharacter: Towards Immersive Role-Playing Agents with Seamless Speech-Language Personality Interaction [123.89581506075461]
  We propose OmniCharacter, a first seamless speech-language personality interaction model to achieve immersive RPAs with low latency.<n> Specifically, OmniCharacter enables agents to consistently exhibit role-specific personality traits and vocal traits throughout the interaction.<n>Our method yields better responses in terms of both content and style compared to existing RPAs and mainstream speech-language models, with a response latency as low as 289ms.
  arXiv  Detail & Related papers  (2025-05-26T17:55:06Z)
• X-TURING: Towards an Enhanced and Efficient Turing Test for Long-Term Dialogue Agents [56.64615470513102]
  The Turing test examines whether AIs exhibit human-like behaviour in natural language conversations.<n>Traditional setting limits each participant to one message at a time and requires constant human participation.<n>This paper proposes textbftextscX-Turing, which enhances the original test with a textitburst dialogue pattern.
  arXiv  Detail & Related papers  (2024-08-19T09:57:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.