Instruct-FD: Can Your Full-Duplex Speech System Follow Turn-Taking Instructions?

• URL: http://arxiv.org/abs/2607.20460v1
• Date: Fri, 15 May 2026 03:02:09 GMT
• Title: Instruct-FD: Can Your Full-Duplex Speech System Follow Turn-Taking Instructions?
• Abstract summary: This is critical in real-world deployments where conversational policies vary across applications.<n>Six state-of-the-art full systems reveals a substantial gap in instruction following turn management: the best model only 64.4% adherence.<n>These findings establish instructionfollowing turn management as a crucial direction for building management management systems.
• Score: 28.523326410588492
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Current full-duplex (FD) spoken dialogue systems can produce fluid interactions, yet it remains unclear whether they can adapt their turn-taking behavior when explicitly instructed. This is critical for real-world deployment, where conversational policies vary across applications (e.g., proactive tutoring vs. passive counseling). We introduce Instruct-FD, an instruction-conditioned benchmark for evaluating controllable turn management in FD systems. To enable this, we develop a human-validated, scalable synthetic pipeline that generates instruction-conditioned conversations, along with a deployment-agnostic multi-turn evaluation protocol and an LLM-based judge. Benchmarking six state-of-the-art full-duplex systems reveals a substantial gap in instruction-following turn management: the best model achieves only 64.4% adherence. Performance is highly uneven across behaviors and scenarios, with proactive behaviors such as model backchanneling and interruption remaining particularly challenging. These findings establish instruction-following turn management as a crucial direction for building adaptable and deployable full-duplex dialogue systems.

Related papers

• DuplexSpeechBench-IFEval: Evaluating Implicit Instruction Following in Full-Duplex Voice Agents [67.07361089429757]
  We measure real-time floor adherence using six real-time speech systems.<n>We find architecture-dependent trade-offs.<n>GPT-Realtime, Mini-o Duplex, Fun-Audio-Chat strongly adhere to persona-consistent content.
  arXiv  Detail & Related papers  (2026-09-03T06:28:02Z)
• S3Gym: Can LLMs Turn Self-Testing and Self-Judging into Self-Improvement? [66.17176838165994]
  Large language models (LLMs) increasingly interact with external environments and accumulate substantial behavioral experience.<n>It remains unclear whether an agent can actively test its behavior, judge the resulting experience, and use that experience to improve future decisions.<n>We introduce textbfStextsuperscript3Gym, an interactive benchmark for evaluating LLM self-improvement.
  arXiv  Detail & Related papers  (2026-08-31T17:05:41Z)
• Facet-Level Persona Control by Trait-Activated Routing with Contrastive SAE for Role-Playing LLMs [6.715533531385597]
  Personality control in Role-Playing Agents (RPAs) is commonly achieved via training-free methods.<n>We propose a contrastive Sparse AutoEncoder framework that learns facet-level personality control vectors aligned with the Big Five 30-facet model.
  arXiv  Detail & Related papers  (2026-02-22T12:39:02Z)
• One Battle After Another: Probing LLMs' Limits on Multi-Turn Instruction Following with a Benchmark Evolving Framework [51.50565654314582]
  Large language models can follow users' instructions throughout a dialogue spanning multiple topics.<n>Existing benchmarks are often limited to a fixed number of turns, making them susceptible to saturation and failing to account for the user's interactive experience.<n>We propose a framework for assessing multi-turn instruction-following ability.
  arXiv  Detail & Related papers  (2025-11-05T14:39:59Z)
• Prompt Decorators: A Declarative and Composable Syntax for Reasoning, Formatting, and Control in LLMs [0.0]
  This paper introduces Prompt Decorators, a declarative, composable syntax that governs behavior through compact control tokens.<n>Each decorator modifies a behavioral dimension, such as verbose reasoning style, structure, or tone, without changing task content.<n>It defines a unified syntax, scoping model, and deterministic processing pipeline enabling predictable and auditable behavior composition.
  arXiv  Detail & Related papers  (2025-10-21T17:35:49Z)
• Data-Model Co-Evolution: Growing Test Sets to Refine LLM Behavior [10.041741229516141]
  Large Language Models (LLMs) allow developers to govern model behavior by editing prompt instructions.<n>We operationalize this paradigm in an interactive system designed to address the challenge of encoding subtle, domain-specific policies into prompt instructions.
  arXiv  Detail & Related papers  (2025-10-14T17:07:37Z)
• Do What? Teaching Vision-Language-Action Models to Reject the Impossible [53.40183895299108]
  Vision-Language-Action (VLA) models have demonstrated strong performance on a range of robotic tasks.<n>We propose Instruct-Verify-and-Act (IVA), a framework that detects when an instruction cannot be executed due to a false premise.<n>Our experiments show that IVA improves false premise detection accuracy by 97.56% over baselines.
  arXiv  Detail & Related papers  (2025-08-22T10:54:33Z)
• Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities [93.09944267871163]
  FullDuplexBench is a benchmark that systematically evaluates key interactive behaviors.<n>By releasing our benchmark code we aim to advance spoken dialogue modeling and the development of more natural and engaging SDMs.
  arXiv  Detail & Related papers  (2025-03-06T18:59:16Z)
• On the Loss of Context-awareness in General Instruction Fine-tuning [101.03941308894191]
  We investigate the loss of context awareness after supervised fine-tuning.<n>We find that the performance decline is associated with a bias toward different roles learned during conversational instruction fine-tuning.<n>We propose a metric to identify context-dependent examples from general instruction fine-tuning datasets.
  arXiv  Detail & Related papers  (2024-11-05T00:16:01Z)
• Inductive-Deductive Strategy Reuse for Multi-Turn Instructional Dialogues [15.959842501166511]
  We propose to explicitly capture the complex rules to help the user simulator pose diverse and in-depth instruction. Experimental results show that our method can generate diverse and in-depth instructions.
  arXiv  Detail & Related papers  (2024-04-17T06:26:32Z)
• DiactTOD: Learning Generalizable Latent Dialogue Acts for Controllable Task-Oriented Dialogue Systems [15.087619144902776]
  We present a novel end-to-end latent dialogue act model (DiactTOD) that represents dialogue acts in a latent space. When pre-trained on a large corpus, DiactTOD is able to predict and control dialogue acts to generate controllable responses.
  arXiv  Detail & Related papers  (2023-08-01T23:29:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.