DuplexSpeechBench-IFEval: Evaluating Implicit Instruction Following in Full-Duplex Voice Agents

• URL: http://arxiv.org/abs/2609.03423v1
• Date: Thu, 03 Sep 2026 06:28:02 GMT
• Title: DuplexSpeechBench-IFEval: Evaluating Implicit Instruction Following in Full-Duplex Voice Agents
• Abstract summary: We measure real-time floor adherence using six real-time speech systems.<n>We find architecture-dependent trade-offs.<n>GPT-Realtime, Mini-o Duplex, Fun-Audio-Chat strongly adhere to persona-consistent content.
• Score: 67.07361089429757
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Full-duplex voice agents must continuously decide when to listen, backchannel, interrupt, handle speech overlaps, take the floor, and yield. Existing benchmarks largely test these behaviors through explicit turn-management instructions, while deployed agents are often configured through roles or personas from which the appropriate conversational behavior must be inferred. We introduce DuplexSpeechBench-IFEval (DSB-IFEval) for evaluating implicit instruction-following in real-time spoken interaction. (DSB-IFEval) comprises 1,038 test cases spanning eight diverse assistant roles and evaluates five conditioning protocols for instruction-following: default behavior, explicit behavioral instructions, persona-implied behavior, combined persona--rule conditioning, and instruction conflict. We measure real-time floor management using a deterministic Instruction Adherence Score (IAS) and persona-consistent content using LLM-judged Persona Adherence Score (PAS). Across six real-time speech systems, we find architecture-dependent trade-offs. Full duplex models like F-Actor and PersonaPlex are more sensitive to whether conversational behavior is stated explicitly or must be inferred from a persona, with adherence dropping by 9.7% and 4.5%, respectively, under persona-only conditioning. In contrast, GPT-Realtime, MiniCPM-o, and Fun-Audio-Chat strongly adhere to persona-consistent content, but their floor behavior does not adapt across explicit and persona-only instructions and remains constrained on several proactive actions. We further find that even if systems reliably follow conflicting directives to their prescribed persona, they still struggle to override them under safety conflict. These results show that inferring the behavior implied by a role, executing it at the appropriate conversational moment, and resolving competing instructions remain distinct challenges for full-duplex voice agents.

Related papers

• The Governance of Human-LLM Interaction: Safety Gating, Civility Steering, and Affective Default Lock-In [0.9558392439655014]
  Large language models (LLMs) increasingly mediate high-stakes interactions in finance, medicine, and mental-health support.<n>We frame interaction style as a governance object: provider-side alignment blocks harmful content, but stabilizes communicative defaults.<n>We show that prompt steerability and regression-to-default are observable indicators of provider control over communicative form.
  arXiv  Detail & Related papers  (2026-06-06T13:36:37Z)
• Instruct-FD: Can Your Full-Duplex Speech System Follow Turn-Taking Instructions? [28.523326410588492]
  This is critical in real-world deployments where conversational policies vary across applications.<n>Six state-of-the-art full systems reveals a substantial gap in instruction following turn management: the best model only 64.4% adherence.<n>These findings establish instructionfollowing turn management as a crucial direction for building management management systems.
  arXiv  Detail & Related papers  (2026-05-15T03:02:09Z)
• PersonaKit (PK): A Plug-and-Play Platform for User Testing Diverse Roles in Full-Duplex Dialogue [8.78029839113787]
  We present PersonaKit, a low-source web platform for rapid prototyping and evaluation of conversational agents.<n>We show that PersonaKit provides an end-to-end framework for studying complex sociolinguistic behaviors in next-generation spoken agents.
  arXiv  Detail & Related papers  (2026-05-07T11:00:03Z)
• KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluation [72.01173512175531]
  KnowU-Bench is an online benchmark for personalized mobile agents.<n>It covers 42 general GUI tasks, 86 personalized tasks, and 64 proactive tasks.<n>Our experiments reveal a striking degradation: agents that excel at explicit task execution fall below 50% under vague instructions.
  arXiv  Detail & Related papers  (2026-04-09T16:50:50Z)
• When Contextual Inference Fails: Cancelability in Interactive Instruction Following [51.2195840589474]
  We introduce Build What I Mean, an interactive benchmark for contextual meaning construction.<n>In BWIM, models must resolve ambiguity by either performing a contextual inference or requesting clarification at a small communication cost.<n>We observe suboptimal strategies, such as partner-blind over-clarification and question-averse guessing under uncertainty.
  arXiv  Detail & Related papers  (2026-03-20T14:46:59Z)
• Facet-Level Persona Control by Trait-Activated Routing with Contrastive SAE for Role-Playing LLMs [6.715533531385597]
  Personality control in Role-Playing Agents (RPAs) is commonly achieved via training-free methods.<n>We propose a contrastive Sparse AutoEncoder framework that learns facet-level personality control vectors aligned with the Big Five 30-facet model.
  arXiv  Detail & Related papers  (2026-02-22T12:39:02Z)
• Persistent Personas? Role-Playing, Instruction Following, and Safety in Extended Interactions [11.415343473837583]
  Persona-assigned large language models (LLMs) are used in domains such as education, healthcare, and sociodemographic simulation.<n>We introduce an evaluation protocol that combines long persona dialogues and evaluation datasets to create dialogue-conditioned benchmarks.<n>We find that persona fidelity degrades over the course of dialogues, especially in goal-oriented conversations.
  arXiv  Detail & Related papers  (2025-12-14T17:27:02Z)
• Learning Steerable Clarification Policies with Collaborative Self-play [67.67872810596839]
  To handle ambiguous queries, AI assistants need a policy for managing their uncertainty.<n>We propose to train steerable policies for managing this uncertainty using self-play.<n>We show this leads to a steerable policy that changes its behavior predictably conditioned on the provided costs.
  arXiv  Detail & Related papers  (2025-12-03T18:49:54Z)
• One Battle After Another: Probing LLMs' Limits on Multi-Turn Instruction Following with a Benchmark Evolving Framework [51.50565654314582]
  Large language models can follow users' instructions throughout a dialogue spanning multiple topics.<n>Existing benchmarks are often limited to a fixed number of turns, making them susceptible to saturation and failing to account for the user's interactive experience.<n>We propose a framework for assessing multi-turn instruction-following ability.
  arXiv  Detail & Related papers  (2025-11-05T14:39:59Z)
• Consistently Simulating Human Personas with Multi-Turn Reinforcement Learning [52.07170679746533]
  Large Language Models (LLMs) are increasingly used to simulate human users in interactive settings such as therapy, education, and social role-play.<n>We introduce a unified framework for evaluating and improving persona consistency in LLM-generated dialogue.<n>We define three automatic metrics: prompt-to-line consistency, line-to-line consistency, and Q&A consistency, that capture different types of persona drift and validate each against human annotations.
  arXiv  Detail & Related papers  (2025-10-31T19:40:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.