RelayS2S: A Dual-Path Speculative Generation for Real-Time Dialogue

• URL: http://arxiv.org/abs/2603.23346v1
• Date: Tue, 24 Mar 2026 15:43:58 GMT
• Title: RelayS2S: A Dual-Path Speculative Generation for Real-Time Dialogue
• Abstract summary: RelayS2S is a hybrid architecture that runs two paths in parallel upon turn detection.<n>A lightweight learned verifier gates the handoff, committing the prefix when appropriate or falling back gracefully to the slow path alone.
• Score: 11.153295574224634
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Real-time spoken dialogue systems face a fundamental tension between latency and response quality. End-to-end speech-to-speech (S2S) models respond immediately and naturally handle turn-taking, backchanneling, and interruption, but produce semantically weaker outputs. Cascaded pipelines (ASR -> LLM) deliver stronger responses at the cost of latency that grows with model size. We present RelayS2S, a hybrid architecture that runs two paths in parallel upon turn detection. The fast path -- a duplex S2S model -- speculatively drafts a short response prefix that is streamed immediately to TTS for low-latency audio onset, while continuing to monitor live audio events. The slow path -- a cascaded ASR -> LLM pipeline -- generates a higher-quality continuation conditioned on the committed prefix, producing a seamless utterance. A lightweight learned verifier gates the handoff, committing the prefix when appropriate or falling back gracefully to the slow path alone. Experiments show that RelayS2S achieves P90 onset latency comparable to the S2S model while retaining 99% cascaded response quality in average score, with benefits growing as the slow-path model scales. Because the prefix handoff requires no architectural modification to either component, RelayS2S serves as a lightweight, drop-in addition to existing cascaded pipelines. Our code and data are publicly available at: https://github.com/mailong25/relays2s

Related papers

• Enabling Streaming User Transcription in Full-Duplex Speech-to-Speech Models [54.83270723528963]
  We propose a method to add streaming capabilities to an existing S2S model.<n>Our approach requires minimal additional parameters and no significant changes to the base S2S model.
  arXiv  Detail & Related papers  (2026-09-14T15:44:18Z)
• Speech Meets ELF: Audio Conditional Continuous-Target Diffusion for Speech Recognition and Translation [48.98480497536377]
  We propose ELF-S2T, an audio-conditioned continuous-target generative model for speech-to-text systems.<n>ELF-S2T processes speech via a frozen Whisper encoder and a single linear projector, prepending the resulting audio condition to the noisy text latent.<n> Experiments on LibriSpeech and CoVoST2 show that ELF-S2T achieves competitive ASR and S2TT performance.
  arXiv  Detail & Related papers  (2026-06-09T03:27:30Z)
• MURMUR: An Efficient Inference System for Long-Form ASR [5.721402685889177]
  Long-form automatic speech recognition requires both high accuracy and low latency.<n>Chunk-based pipelines process audio in parallel windows for low latency, but lose cross-context context.<n>Long-context models resolve everything in a single pass for better accuracy, but are an order of magnitude slower.<n>We propose Murmur, an inference system that overcomes this trade-off by operating at two levels.
  arXiv  Detail & Related papers  (2026-05-31T22:54:57Z)
• Stream2LLM: Overlap Context Streaming and Prefill for Reduced Time-to-First-Token (TTFT) [2.8350851063316873]
  Streaming context incrementally--overlapping retrieval with inference--can mitigate this latency.<n>We present Stream2LLM, a streaming-aware LLM serving system for concurrent prefill-decode disaggregated deployments.
  arXiv  Detail & Related papers  (2026-03-29T06:49:12Z)
• LTS-VoiceAgent: A Listen-Think-Speak Framework for Efficient Streaming Voice Interaction via Semantic Triggering and Incremental Reasoning [27.13598270494417]
  LTS-VoiceAgent is a Listen-Think-Speak framework that separates when to think from how to reason incrementally.<n>It features a Dynamic Semantic Trigger to detect meaningful prefixes, and a Dual-Role Stream Orchestrator that coordinates a background Thinker and a foreground Speaker.
  arXiv  Detail & Related papers  (2026-01-26T15:42:35Z)
• Chain-of-Thought Reasoning in Streaming Full-Duplex End-to-End Spoken Dialogue Systems [82.70507055599093]
  We propose a Streaming Chain-of-Thought (CoT) framework for Duplex SDS.<n>We create intermediate targets-aligned user transcripts and system responses for each block.<n>Experiments show that our approach produces more coherent and interpretable responses than existing duplex methods.
  arXiv  Detail & Related papers  (2025-10-02T14:33:05Z)
• Stream RAG: Instant and Accurate Spoken Dialogue Systems with Streaming Tool Usage [66.67531241554546]
  End-to-end speech-in speech-out dialogue systems are emerging as a powerful alternative to traditional ASR-LLM-TTS pipelines.<n>We introduce the first approach to extend tool use directly into speech-in speech-out systems.<n>We propose Streaming Retrieval-Augmented Generation (Streaming RAG), a novel framework that reduces user-perceived latency by predicting tool queries in parallel with user speech.
  arXiv  Detail & Related papers  (2025-10-02T14:18:20Z)
• KAME: Tandem Architecture for Enhancing Knowledge in Real-Time Speech-to-Speech Conversational AI [14.667102744113295]
  Real-time speech-to-speech (S2S) models excel at generating low-latency conversational responses but often lack deep knowledge and semantic understanding.<n>C cascaded systems combining automatic speech recognition, a text-based Large Language Model (LLM), and text-to-speech synthesis offer superior knowledge representation at the cost of high latency.<n>This paper introduces a novel hybrid architecture that bridges the gap between these two paradigms.
  arXiv  Detail & Related papers  (2025-09-26T00:46:34Z)
• Whisfusion: Parallel ASR Decoding via a Diffusion Transformer [7.327454599174306]
  Whisfusion is a framework to fuse a pre-trained Whisper encoder with a text diffusion decoder.<n>A lightweight cross-attention adapter trained via parameter-efficient fine-tuning (PEFT) bridges the two modalities.<n>Fine-tuned solely on LibriSpeech (960h), Whisfusion achieves a lower WER than Whisper-tiny, and offers comparable latency on short audio.
  arXiv  Detail & Related papers  (2025-08-09T17:20:54Z)
• Pseudo-Autoregressive Neural Codec Language Models for Efficient Zero-Shot Text-to-Speech Synthesis [64.12708207721276]
  We introduce a novel pseudo-autoregressive (PAR) language modeling approach that unifies AR and NAR modeling.<n>Building on PAR, we propose PALLE, a two-stage TTS system that leverages PAR for initial generation followed by NAR refinement.<n>Experiments demonstrate that PALLE, trained on LibriTTS, outperforms state-of-the-art systems trained on large-scale data.
  arXiv  Detail & Related papers  (2025-04-14T16:03:21Z)
• V2SFlow: Video-to-Speech Generation with Speech Decomposition and Rectified Flow [57.51550409392103]
  We introduce V2SFlow, a novel Video-to-Speech (V2S) framework designed to generate natural and intelligible speech directly from silent talking face videos.<n>To address these challenges, we decompose the speech signal into manageable subspaces, each representing distinct speech attributes, and predict them directly from the visual input.<n>To generate coherent and realistic speech from these predicted attributes, we employ a rectified flow matching decoder built on a Transformer architecture.
  arXiv  Detail & Related papers  (2024-11-29T05:55:20Z)
• Turbocharge Speech Understanding with Pilot Inference [0.9699101045941684]
  This paper sets to accelerate modern speech understanding on resource-constrained edge devices. It takes a hybrid approach: to speed up on-device execution; to offload inputs that are beyond the device's capacity. Our prototype, called PASU, is tested on Arm platforms with 6 - 8 cores: it delivers SOTA accuracy; it reduces the end-to-end latency by 2x and reduces the offloading needs by 2x.
  arXiv  Detail & Related papers  (2023-11-22T17:14:18Z)
• Delay-penalized transducer for low-latency streaming ASR [26.39851372961386]
  We propose a simple way to penalize symbol delay in transducer model, so that we can balance the trade-off between symbol delay and accuracy for streaming models without external alignments. Our method achieves similar delay-accuracy trade-off to the previously published FastEmit, but we believe our method is preferable because it has a better justification: it is equivalent to penalizing the average symbol delay.
  arXiv  Detail & Related papers  (2022-10-31T07:03:50Z)
• FastLTS: Non-Autoregressive End-to-End Unconstrained Lip-to-Speech Synthesis [77.06890315052563]
  We propose FastLTS, a non-autoregressive end-to-end model which can directly synthesize high-quality speech audios from unconstrained talking videos with low latency. Experiments show that our model achieves $19.76times$ speedup for audio generation compared with the current autoregressive model on input sequences of 3 seconds.
  arXiv  Detail & Related papers  (2022-07-08T10:10:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.