MoshiRAG: Asynchronous Knowledge Retrieval for Full-Duplex Speech Language Models

• URL: http://arxiv.org/abs/2604.12928v2
• Date: Fri, 17 Apr 2026 07:00:02 GMT
• Title: MoshiRAG: Asynchronous Knowledge Retrieval for Full-Duplex Speech Language Models
• Abstract summary: Asynchronous full-speech models are distinguished by their full-time interactivity and naturalness of AI pauses.<n>Our framework enables the model to identify knowledge-demand conversational queries and ground responses in external information.<n>Our design supports plug-and-play retrieval methods without retraining and demonstrates strong performance on out-of-toe reasoning tasks.
• Score: 62.05118198431989
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Speech-to-speech language models have recently emerged to enhance the naturalness of conversational AI. In particular, full-duplex models are distinguished by their real-time interactivity, including handling of pauses, interruptions, and backchannels. However, improving their factuality remains an open challenge. While scaling the model size could address this gap, it would make real-time inference prohibitively expensive. In this work, we propose MoshiRAG, a modular approach that combines a compact full-duplex interface with selective retrieval to access more powerful knowledge sources. Our asynchronous framework enables the model to identify knowledge-demanding queries and ground its responses in external information. By leveraging the natural temporal gap between response onset and the delivery of core information, the retrieval process can be completed while maintaining a natural conversation flow. With this approach, MoshiRAG achieves factuality comparable to the best publicly released non-duplex speech language models while preserving the interactivity inherent to full-duplex systems. Moreover, our flexible design supports plug-and-play retrieval methods without retraining and demonstrates strong performance on out-of-domain mathematical reasoning tasks.

Related papers

• Multi-Faceted Interactivity Alignment in Full-Duplex Speech Models [53.470209949659115]
  We propose a post-training alignment method that improves full spoken dialogue models through RL.<n>We address the four canonical axes of interactivity: handling turn-taking, back-channel pause, and user interruption.<n>An extra-based reward for response quality prevents semantic degradation.
  arXiv  Detail & Related papers  (2026-06-09T17:46:55Z)
• Sommelier: Scalable Open Multi-turn Audio Pre-processing for Full-duplex Speech Language Models [48.29876406999829]
  There is growing demand for AI systems capable of real-time human-computer interaction.<n>Large-scale resources are singlespeakers or limited volume.<n>We present robust and scalable open-source processing pipeline designed for full-source model development.
  arXiv  Detail & Related papers  (2026-03-20T09:10:43Z)
• FLEXI: Benchmarking Full-duplex Human-LLM Speech Interaction [49.83226596963294]
  Speech-computer human interaction enables real-time spoken dialogue systems.<n>Modelling and benchmarking these models remains a fundamental challenge.<n>We introduce FLEXI, the first benchmark for full-human spoken interaction.
  arXiv  Detail & Related papers  (2025-09-26T11:57:42Z)
• KAME: Tandem Architecture for Enhancing Knowledge in Real-Time Speech-to-Speech Conversational AI [14.667102744113295]
  Real-time speech-to-speech (S2S) models excel at generating low-latency conversational responses but often lack deep knowledge and semantic understanding.<n>C cascaded systems combining automatic speech recognition, a text-based Large Language Model (LLM), and text-to-speech synthesis offer superior knowledge representation at the cost of high latency.<n>This paper introduces a novel hybrid architecture that bridges the gap between these two paradigms.
  arXiv  Detail & Related papers  (2025-09-26T00:46:34Z)
• UniConv: Unifying Retrieval and Response Generation for Large Language Models in Conversations [71.79210031338464]
  We show how to unify dense retrieval and response generation for large language models in conversation.<n>We conduct joint fine-tuning with different objectives and design two mechanisms to reduce the inconsistency risks.<n>The evaluations on five conversational search datasets demonstrate that our unified model can mutually improve both tasks and outperform the existing baselines.
  arXiv  Detail & Related papers  (2025-07-09T17:02:40Z)
• Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities [93.09944267871163]
  FullDuplexBench is a benchmark that systematically evaluates key interactive behaviors.<n>By releasing our benchmark code we aim to advance spoken dialogue modeling and the development of more natural and engaging SDMs.
  arXiv  Detail & Related papers  (2025-03-06T18:59:16Z)
• OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation [53.7173034249361]
  End-to-end GPT-based model OmniFlatten capable of effectively modeling complex behaviors inherent natural conversations with low latency.<n>Our approach offers a simple modeling technique and a promising research direction for developing efficient and natural end-to-end full- spoken dialogue systems.
  arXiv  Detail & Related papers  (2024-10-23T11:58:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.