Synchronization and Turn-Taking in Full-Duplex Speech Dialogue Models

• URL: http://arxiv.org/abs/2605.20356v1
• Date: Tue, 19 May 2026 18:11:03 GMT
• Title: Synchronization and Turn-Taking in Full-Duplex Speech Dialogue Models
• Abstract summary: Full spoken dialogue models can speak simultaneously closer to human conversation than turn-based systems.<n>We study such models coordinate their internal interaction using neural coupling in human communication.<n>We find strong representational synchronization under no noise conditions, peaking near zero lag and degrading noise.
• Score: 3.5946669116828134
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Full-duplex spoken dialogue models (SDMs) can listen and speak simultaneously, enabling interaction dynamics closer to human conversation than turn-based systems. Inspired by neural coupling in human communication, we study how such models coordinate their internal representations during interaction. We simulate full-duplex dialogues between two instances of the pretrained \textit{Moshi} model under controlled conditions, manipulating channel noise and decoding bias. Synchronization is measured using Centered Kernel Alignment (CKA) across temporal lags, while anticipatory turn-taking cues are probed from delayed internal activations using causal LSTM models, from both speaker and listener perspectives. We find strong representational synchronization under no noise conditions, peaking near zero lag and degrading with noise, and we show that internal states encode anticipatory information that supports turn-taking prediction ahead of time.

Related papers

• Overcoming State Inertia in Full-Duplex Spoken Language Models via Activation Steering [87.43777061308658]
  We analyze predictive behavior in full spoken language models (FDSLMs)<n>We show that FDSLMs dynamically modulate their internal focus between two states: a generative state aligned with model output generation and a perceptive state aligned incoming user input.<n>During user interruptions, the model remains transiently biased toward the generative state before transitioning into the perceptive state, causing it miss the beginning of incoming input.
  arXiv  Detail & Related papers  (2026-06-09T19:08:07Z)
• Multi-Faceted Interactivity Alignment in Full-Duplex Speech Models [53.470209949659115]
  We propose a post-training alignment method that improves full spoken dialogue models through RL.<n>We address the four canonical axes of interactivity: handling turn-taking, back-channel pause, and user interruption.<n>An extra-based reward for response quality prevents semantic degradation.
  arXiv  Detail & Related papers  (2026-06-09T17:46:55Z)
• U-Mind: A Unified Framework for Real-Time Multimodal Interaction with Audiovisual Generation [48.6868174403074]
  We introduce U-Mind, the first unified system for high-intelligence multimodal dialogue.<n>It supports real-time generation and jointly models language, speech, motion, and video synthesis within a single interactive loop.<n>We show that U-Mind achieves state-of-the-art performance on a range of multimodal interaction tasks.
  arXiv  Detail & Related papers  (2026-02-27T07:07:02Z)
• Chronological Thinking in Full-Duplex Spoken Dialogue Language Models [66.84843878538207]
  Chronological Thinking aims to improve response quality in full SDLMs.<n>No additional latency: once the user stops speaking, the agent halts thinking and begins speaking without further delay.<n>Results: Experiments demonstrate the effectiveness of chronological thinking through both objective metrics and human evaluations.
  arXiv  Detail & Related papers  (2025-10-02T10:28:11Z)
• FLEXI: Benchmarking Full-duplex Human-LLM Speech Interaction [49.83226596963294]
  Speech-computer human interaction enables real-time spoken dialogue systems.<n>Modelling and benchmarking these models remains a fundamental challenge.<n>We introduce FLEXI, the first benchmark for full-human spoken interaction.
  arXiv  Detail & Related papers  (2025-09-26T11:57:42Z)
• From Turn-Taking to Synchronous Dialogue: A Survey of Full-Duplex Spoken Language Models [12.741006204459637]
  Full-Duplex voice communication enables simultaneous listening and speaking with natural turn-taking, overlapping speech, and interruptions.<n>This survey comprehensively reviews Full-Duplex Spoken Language Models (FD-SLMs)<n>We identify fundamental challenges: synchronous data scarcity, architectural divergence, and evaluation gaps.
  arXiv  Detail & Related papers  (2025-09-18T01:00:58Z)
• Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities [93.09944267871163]
  FullDuplexBench is a benchmark that systematically evaluates key interactive behaviors.<n>By releasing our benchmark code we aim to advance spoken dialogue modeling and the development of more natural and engaging SDMs.
  arXiv  Detail & Related papers  (2025-03-06T18:59:16Z)
• Yeah, Un, Oh: Continuous and Real-time Backchannel Prediction with Fine-tuning of Voice Activity Projection [24.71649541757314]
  Short backchannel utterances such as "yeah" and "oh" play a crucial role in facilitating smooth and engaging dialogue.<n>This paper proposes a novel method for real-time, continuous backchannel prediction using a fine-tuned Voice Activity Projection model.
  arXiv  Detail & Related papers  (2024-10-21T11:57:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.