Sommelier: Scalable Open Multi-turn Audio Pre-processing for Full-duplex Speech Language Models

• URL: http://arxiv.org/abs/2603.25750v1
• Date: Fri, 20 Mar 2026 09:10:43 GMT
• Title: Sommelier: Scalable Open Multi-turn Audio Pre-processing for Full-duplex Speech Language Models
• Abstract summary: There is growing demand for AI systems capable of real-time human-computer interaction.<n>Large-scale resources are singlespeakers or limited volume.<n>We present robust and scalable open-source processing pipeline designed for full-source model development.
• Score: 48.29876406999829
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As the paradigm of AI shifts from text-based LLMs to Speech Language Models (SLMs), there is a growing demand for full-duplex systems capable of real-time, natural human-computer interaction. However, the development of such models is constrained by the scarcity of high-quality, multi-speaker conversational data, as existing large-scale resources are predominantly single-speaker or limited in volume. Addressing the complex dynamics of natural dialogue, such as overlapping and back-channeling remains a challenge, with standard processing pipelines suffering from diarization errors and ASR hallucinations. To bridge this gap, we present a robust and scalable open-source data processing pipeline designed for full-duplex model.

Related papers

• MoshiRAG: Asynchronous Knowledge Retrieval for Full-Duplex Speech Language Models [62.05118198431989]
  Asynchronous full-speech models are distinguished by their full-time interactivity and naturalness of AI pauses.<n>Our framework enables the model to identify knowledge-demand conversational queries and ground responses in external information.<n>Our design supports plug-and-play retrieval methods without retraining and demonstrates strong performance on out-of-toe reasoning tasks.
  arXiv  Detail & Related papers  (2026-04-14T16:17:52Z)
• Covo-Audio Technical Report [61.09708870154148]
  Covo-Audio, a 7B-end LALM, directly processes continuous audio inputs and generates audio outputs within a single unified architecture.<n>Covo-Audio-Chat, a dialogue-oriented variant, demonstrates semantic strong spoken conversational abilities.
  arXiv  Detail & Related papers  (2026-02-10T14:31:11Z)
• ChatUMM: Robust Context Tracking for Conversational Interleaved Generation [44.19929499646892]
  Unified multimodal models (UMMs) have achieved remarkable progress yet remain constrained by a single-turn interaction paradigm.<n>We present ChatUMM, a conversational unified model that excels at robust context tracking to sustain interleaved multimodal generation.<n>ChatUMM derives its capabilities from an interleaved multi-turn training strategy that models serialized text-image streams as a continuous conversational flow.
  arXiv  Detail & Related papers  (2026-02-06T07:11:50Z)
• HI-TransPA: Hearing Impairments Translation Personal Assistant [23.33416647487016]
  We introduce the Omni-Model paradigm into assistive technology and present HI-TransPA, an instruction-driven audio-visual personal assistant.<n>The model fuses indistinct speech with lip dynamics, enabling both translation and dialogue within a single multimodal framework.<n>Experiments on purpose-built HI-Dialogue dataset show that HI-TransPA achieves state-of-the-art performance in both literal accuracy and semantic fidelity.
  arXiv  Detail & Related papers  (2025-11-13T03:27:39Z)
• FLEXI: Benchmarking Full-duplex Human-LLM Speech Interaction [49.83226596963294]
  Speech-computer human interaction enables real-time spoken dialogue systems.<n>Modelling and benchmarking these models remains a fundamental challenge.<n>We introduce FLEXI, the first benchmark for full-human spoken interaction.
  arXiv  Detail & Related papers  (2025-09-26T11:57:42Z)
• SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversation [17.56310064245171]
  SALMON-N-omni is the first single standalone full-byte speech LLM that operates without its token transition backbone.<n>It features a novel dynamic thinking mechanism within the LLM backbone, enabling the model to learn when between speaking and listening.<n> SALMON-N-omni demonstrates strong performance in complex conversational scenarios, including turn-taking, backchanneling echo cancellation and context-dependent bargein.
  arXiv  Detail & Related papers  (2025-05-17T08:13:59Z)
• Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities [93.09944267871163]
  FullDuplexBench is a benchmark that systematically evaluates key interactive behaviors.<n>By releasing our benchmark code we aim to advance spoken dialogue modeling and the development of more natural and engaging SDMs.
  arXiv  Detail & Related papers  (2025-03-06T18:59:16Z)
• SEAL: Speech Embedding Alignment Learning for Speech Large Language Model with Retrieval-Augmented Generation [10.828717295018123]
  We propose a unified embedding framework that eliminates the need for intermediate text representations.<n>Our model reduces pipeline latency by 50% while achieving higher retrieval accuracy compared to traditional two-stage methods.
  arXiv  Detail & Related papers  (2025-01-26T15:04:02Z)
• OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation [53.7173034249361]
  End-to-end GPT-based model OmniFlatten capable of effectively modeling complex behaviors inherent natural conversations with low latency.<n>Our approach offers a simple modeling technique and a promising research direction for developing efficient and natural end-to-end full- spoken dialogue systems.
  arXiv  Detail & Related papers  (2024-10-23T11:58:58Z)
• Generating Data with Text-to-Speech and Large-Language Models for Conversational Speech Recognition [48.527630771422935]
  We propose a synthetic data generation pipeline for multi-speaker conversational ASR. We conduct evaluation by fine-tuning the Whisper ASR model for telephone and distant conversational speech settings.
  arXiv  Detail & Related papers  (2024-08-17T14:47:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.