BayLing-Duplex: Native Full-Duplex Speech Dialogue with a Single Autoregressive LLM

• URL: http://arxiv.org/abs/2606.14528v1
• Date: Fri, 12 Jun 2026 15:01:51 GMT
• Title: BayLing-Duplex: Native Full-Duplex Speech Dialogue with a Single Autoregressive LLM
• Abstract summary: BayLing-Duplex is a native full-generation SpeechLM that decides where to listen, when to speak, and sacrifice when to stop with no auxiliary turn-taking module.<n>It reaches 92% turn-taking success and 100% interruption success on InstructS2S-Eval, while improving speech-response score from 2.17 to 3.39 over Moshi.
• Score: 37.246890931249496
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Real-time, full-duplex speech interaction is a key feature of next-generation spoken chatbots, allowing the model to listen and speak at the same time and to handle natural phenomena such as overlap, hesitation, and barge-in. Existing speech language models (SpeechLMs) such as LLaMA-Omni and GLM-4-Voice are still turn-based and rely on an external Voice Activity Detection (VAD) module to mark the end of the user's turn, which fundamentally limits their interactive ability. In this paper, we introduce BayLing-Duplex, a native full-duplex SpeechLM where a single autoregressive LLM decides when to listen, when to speak, and when to stop, with no auxiliary turn-taking module. The design adds only a few special tokens to the standard vocabulary, so it transfers across LLMs and reuses existing training and serving stacks with no architectural adaptation. Starting from the public GLM-4-Voice checkpoint and using only 400K full-duplex samples for fine-tuning followed by a lightweight DPO stage, BayLing-Duplex reaches 92% turn-taking success and 100% interruption success on InstructS2S-Eval, while improving the speech-response score from 2.17 to 3.39 over Moshi. BayLing-Duplex also matches or surpasses its turn-based counterpart on Llama Questions, Web Questions, and Alpaca-Eval, showing that simultaneous listen-and-speak modeling does not sacrifice response quality.

Related papers

• Enabling Proactive Spoken Turns via a Generalized Style-Aware Full-Duplex Framework [51.969792455913655]
  2 half dialogue systems system system waits for user turn completion before responds.<n>This creates key challenge: anticipate turn timing without sacrificing response quality.<n>We present WildTurnTurn, a largescale, real-world English dataset containing approximately981 hours of multi-turn conversations from face-to-face and telephone conversations, with five turn-level interaction styles, trained on WildTurn LPS-TC rich spoken dynamics that exhibit rich spoken dynamics that are not captured by existing static benchmarks, including we introduce a two-tier evaluation scheme that assesses both in-level timing precision and turn-level interaction quality under realistic streaming constraints,
  arXiv  Detail & Related papers  (2026-08-04T08:22:50Z)
• Multi-Faceted Interactivity Alignment in Full-Duplex Speech Models [53.470209949659115]
  We propose a post-training alignment method that improves full spoken dialogue models through RL.<n>We address the four canonical axes of interactivity: handling turn-taking, back-channel pause, and user interruption.<n>An extra-based reward for response quality prevents semantic degradation.
  arXiv  Detail & Related papers  (2026-06-09T17:46:55Z)
• Liberating LLM Capabilities in Full-Duplex Speech Models [7.252793405548695]
  Speech-based large modality language models are typically constrained by spoken replies.<n>We propose a textWritefirst tri-channel paradigm in which a single autore listens to user audio, writes visible freeform text as its primary output, and what speaks realtime oral response in parallel under shared causal attention.<n>These results suggest visible writing can serve as a first-class output channel for speech interaction without sacrificing realtime oral response.
  arXiv  Detail & Related papers  (2026-05-04T17:54:41Z)
• DuplexCascade: Full-Duplex Speech-to-Speech Dialogue with VAD-Free Cascaded ASR-LLM-TTS Pipeline and Micro-Turn Optimization [17.80344425747229]
  Our key idea is to convert conventional utterance-wise long turns into chunk-turn interactions.<n>To reliably coordinate turn-taking response timing, we introduce a set conversational special control tokens.
  arXiv  Detail & Related papers  (2026-03-10T04:35:22Z)
• Fun-Audio-Chat Technical Report [71.07966678560291]
  temporal resolution between speech tokens (25Hz) and text tokens (3Hz) mitigates semantic information mismatchs and incurs high computational costs.<n>We introduce Fun-Audio-Chat, a Large-stage Spoken Speech-to-scale tasks.<n>Fun-Audio-Chat 8B and MoE 30BA3B achieve competitive performance on SpeechText and Speech-to-scale tasks.
  arXiv  Detail & Related papers  (2025-12-23T08:35:27Z)
• A Unified Speech LLM for Diarization and Speech Recognition in Multilingual Conversations [25.58593495281218]
  We propose a unified speech LLM that jointly performs diarization and ASR in an end-to-end manner.<n>By reformulating the training data format and modifying the inference procedure, our model addresses the ambiguity inherent in pre-segmented audio.
  arXiv  Detail & Related papers  (2025-06-26T01:54:02Z)
• DrVoice: Parallel Speech-Text Voice Conversation Model via Dual-Resolution Speech Representations [62.00227663434538]
  DRVOICE-7B establishes new state-of-the-art (SOTA) on OpenAudioBench and Big Bench Audio benchmarks.<n>This paper presents DrVoice, a parallel speech-text voice conversation model based on joint autoregressive modeling.
  arXiv  Detail & Related papers  (2025-06-11T02:57:22Z)
• SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversation [17.56310064245171]
  SALMON-N-omni is the first single standalone full-byte speech LLM that operates without its token transition backbone.<n>It features a novel dynamic thinking mechanism within the LLM backbone, enabling the model to learn when between speaking and listening.<n> SALMON-N-omni demonstrates strong performance in complex conversational scenarios, including turn-taking, backchanneling echo cancellation and context-dependent bargein.
  arXiv  Detail & Related papers  (2025-05-17T08:13:59Z)
• OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation [53.7173034249361]
  End-to-end GPT-based model OmniFlatten capable of effectively modeling complex behaviors inherent natural conversations with low latency.<n>Our approach offers a simple modeling technique and a promising research direction for developing efficient and natural end-to-end full- spoken dialogue systems.
  arXiv  Detail & Related papers  (2024-10-23T11:58:58Z)
• Beyond the Turn-Based Game: Enabling Real-Time Conversations with Duplex Models [66.24055500785657]
  Traditional turn-based chat systems prevent users from verbally interacting with system while it is generating responses. To overcome these limitations, we adapt existing LLMs to listen users while generating output and provide users with instant feedback. We build a dataset consisting of alternating time slices of queries and responses as well as covering typical feedback types in instantaneous interactions.
  arXiv  Detail & Related papers  (2024-06-22T03:20:10Z)
• A Full-duplex Speech Dialogue Scheme Based On Large Language Models [23.994130020644842]
  generative generative dialogue system capable in a full manner allowing seamless interaction. System generates tokens for inquiry responses and makes autonomous decisions to respond to wait for, or operate the user.
  arXiv  Detail & Related papers  (2024-05-29T20:05:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.