CleanS2S: Single-file Framework for Proactive Speech-to-Speech Interaction

• URL: http://arxiv.org/abs/2506.01268v1
• Date: Mon, 02 Jun 2025 02:40:46 GMT
• Title: CleanS2S: Single-file Framework for Proactive Speech-to-Speech Interaction
• Abstract summary: CleanS2S is a framework human-like speech interaction that advances conversational AI single-file implementation and proactive dialogue capabilities.<n>Our system integrates automatic speech recognition language models, and text-to-speech synthesis into a unified pipeline with real-time interruption.
• Score: 2.854461601795248
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: CleanS2S is a framework for human-like speech-to-speech interaction that advances conversational AI through single-file implementation and proactive dialogue capabilities. Our system integrates automatic speech recognition, large language models, and text-to-speech synthesis into a unified pipeline with real-time interruption handling, achieving low transition latency through full-duplex websocket connections and non-blocking I/O. Beyond conventional chatbot paradigms, we pioneer a proactive interaction mechanism, which combines memory systems with Subjective Action Judgement module, enabling five human-like response strategies: interruption, refusal, deflection, silence, and standard response. The memory module dynamically aggregates historical, and contextual data to inform interaction decisions. This approach breaks the rigid turn-based convention by allowing system-initiated dialog control and context-aware response selection. And we propose Action Judgement SFT that assesses input streams for responses strategies. The framework's single-file implementation with atomic configurations offers researchers unprecedented transparency and extensibility for interaction agents. The code of CleanS2S is released at \https://github.com/opendilab/CleanS2S.

Related papers

• Noise Adaptive Streaming Audio-Visual Speech Token Enhancement for Robust Full-Duplex Spoken Dialogue Models [49.372283081631366]
  AVSTE restores semantic speech tokens from noisy audio and lip video before they reach speech perception.<n>When integrated with Moshi, AVSTE improves average GPT-4ojudged response coherence from 1.42 1.91 under same-data-set speaker interference.
  arXiv  Detail & Related papers  (2026-09-08T07:59:13Z)
• PRISM: Prosody-Integrated Multi-Agent Reasoning Framework for Empathetic Spoken Dialogue [29.006038821337327]
  PRISM is a multi-agent framework for empathetic spoken dialogue.<n>It decouples speech perception, response generation, and speech synthesis into coordinated components.
  arXiv  Detail & Related papers  (2026-06-11T04:59:33Z)
• Multi-Faceted Interactivity Alignment in Full-Duplex Speech Models [53.470209949659115]
  We propose a post-training alignment method that improves full spoken dialogue models through RL.<n>We address the four canonical axes of interactivity: handling turn-taking, back-channel pause, and user interruption.<n>An extra-based reward for response quality prevents semantic degradation.
  arXiv  Detail & Related papers  (2026-06-09T17:46:55Z)
• UAF: A Unified Audio Front-end LLM for Full-Duplex Speech Interaction [7.775050285048427]
  Speech assistant is driving artificial propagation toward human-like conversational systems.<n>Front-end components, such as voice activity detection (VAD) and turn-taking detection (TD) are crucial for speech assistant.<n>We propose the first unified audio frontend LLM (UAF) tailored for fullgressive speech systems.
  arXiv  Detail & Related papers  (2026-04-21T08:24:55Z)
• Unit-Based Agent for Semi-Cascaded Full-Duplex Dialogue Systems [17.54500572999039]
  Full voice interaction is process of natural human computer interaction.<n>This framework framework synthesises complex dialogue into minimal conversational units.<n>System operates in a train-free, plug-play manner.
  arXiv  Detail & Related papers  (2026-01-28T04:00:37Z)
• Chronological Thinking in Full-Duplex Spoken Dialogue Language Models [66.84843878538207]
  Chronological Thinking aims to improve response quality in full SDLMs.<n>No additional latency: once the user stops speaking, the agent halts thinking and begins speaking without further delay.<n>Results: Experiments demonstrate the effectiveness of chronological thinking through both objective metrics and human evaluations.
  arXiv  Detail & Related papers  (2025-10-02T10:28:11Z)
• MOSS-Speech: Towards True Speech-to-Speech Models Without Text Guidance [66.74042564585942]
  MOSS-Speech is a true speech-to-speech large language model that directly understands and generates speech without relying on text guidance.<n>Our work establishes a new paradigm for expressive and efficient end-to-end speech interaction.
  arXiv  Detail & Related papers  (2025-10-01T04:32:37Z)
• KAME: Tandem Architecture for Enhancing Knowledge in Real-Time Speech-to-Speech Conversational AI [14.667102744113295]
  Real-time speech-to-speech (S2S) models excel at generating low-latency conversational responses but often lack deep knowledge and semantic understanding.<n>C cascaded systems combining automatic speech recognition, a text-based Large Language Model (LLM), and text-to-speech synthesis offer superior knowledge representation at the cost of high latency.<n>This paper introduces a novel hybrid architecture that bridges the gap between these two paradigms.
  arXiv  Detail & Related papers  (2025-09-26T00:46:34Z)
• Towards Efficient Speech-Text Jointly Decoding within One Speech Language Model [76.06585781346601]
  Speech language models (Speech LMs) enable end-to-end speech-text modelling within a single model.<n>The choice of speech-text jointly decoding paradigm plays a critical role in performance, efficiency, and alignment quality.
  arXiv  Detail & Related papers  (2025-06-04T23:53:49Z)
• OmniResponse: Online Multimodal Conversational Response Generation in Dyadic Interactions [62.19092662469285]
  Online Multimodal Conversational Response Generation (OMCRG) is a novel task designed to produce synchronized verbal and non-verbal listener feedback online.<n>We propose OmniResponse, a Multimodal Large Language Model (MLLM) that autoregressively generates accurate multimodal listener responses.<n>We offer ResponseNet, a dataset of 696 detailed dyadic interactions featuring synchronized split-screen videos, multichannel audio, transcripts, and annotated facial behaviors.
  arXiv  Detail & Related papers  (2025-05-27T20:12:46Z)
• A Multi-view Discourse Framework for Integrating Semantic and Syntactic Features in Dialog Agents [0.0]
  Multiturn dialogue models aim to generate human-like responses by leveraging conversational context.<n>Existing methods often neglect the interactions between these utterances or treat all of them as equally significant.<n>This paper introduces a discourse-aware framework for response selection in retrieval-based dialogue systems.
  arXiv  Detail & Related papers  (2025-04-12T04:22:18Z)
• Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities [93.09944267871163]
  FullDuplexBench is a benchmark that systematically evaluates key conversational behaviors.<n>We aim to advance spoken dialogue modeling and encourage the development of more interactive and natural dialogue systems.
  arXiv  Detail & Related papers  (2025-03-06T18:59:16Z)
• Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reaction [81.34648970317383]
  We present Dispider, a system that disentangles Perception, Decision, and Reaction.<n>Experiments show that Dispider not only maintains strong performance in conventional video QA tasks, but also significantly surpasses previous online models in streaming scenario responses.
  arXiv  Detail & Related papers  (2025-01-06T18:55:10Z)
• OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation [53.7173034249361]
  End-to-end GPT-based model OmniFlatten capable of effectively modeling complex behaviors inherent natural conversations with low latency.<n>Our approach offers a simple modeling technique and a promising research direction for developing efficient and natural end-to-end full- spoken dialogue systems.
  arXiv  Detail & Related papers  (2024-10-23T11:58:58Z)
• Moshi: a speech-text foundation model for real-time dialogue [78.88479749811376]
  Current systems for spoken dialogue rely on pipelines independent voice activity detection and text-to-speech. We show how Moshi Moshi can provide streaming speech recognition and text-to-speech. Our resulting model is first real-time full spoken large language model modality.
  arXiv  Detail & Related papers  (2024-09-17T17:55:39Z)
• Style-Talker: Finetuning Audio Language Model and Style-Based Text-to-Speech Model for Fast Spoken Dialogue Generation [16.724603503894166]
  Style-Talker is an innovative framework that fine-tunes an audio LLM alongside a style-based TTS model for fast spoken dialog generation. Our experimental results show that Style-Talker significantly outperforms the conventional cascade and speech-to-speech baselines in terms of both dialogue naturalness and coherence.
  arXiv  Detail & Related papers  (2024-08-13T04:35:11Z)
• A Non-autoregressive Generation Framework for End-to-End Simultaneous Speech-to-Speech Translation [48.84039953531355]
  We propose a novel non-autoregressive generation framework for simultaneous speech translation (NAST-S2X) NAST-S2X integrates speech-to-text and speech-to-speech tasks into a unified end-to-end framework. It achieves high-quality simultaneous interpretation within a delay of less than 3 seconds and provides a 28 times decoding speedup in offline generation.
  arXiv  Detail & Related papers  (2024-06-11T04:25:48Z)
• Turn-Taking Prediction for Natural Conversational Speech [40.189938418201656]
  A common conversational utterance often involves multiple queries with turn-taking. Disfluencies include pausing to think, hesitations, word lengthening, filled pauses and repeated phrases. We present a turntaking predictor built on top of the end-to-end (E2E) speech recognizer.
  arXiv  Detail & Related papers  (2022-08-29T01:09:23Z)
• An Approach to Inference-Driven Dialogue Management within a Social Chatbot [10.760026478889667]
  Instead of framing conversation as a sequence of response generation tasks, we model conversation as a collaborative inference process. Our pipeline accomplishes this modelling in three broad stages. This approach lends itself to understanding latent semantics of user inputs, flexible initiative taking, and responses that are novel and coherent with the dialogue context.
  arXiv  Detail & Related papers  (2021-10-31T19:01:07Z)
• Smoothing Dialogue States for Open Conversational Machine Reading [70.83783364292438]
  We propose an effective gating strategy by smoothing the two dialogue states in only one decoder and bridge decision making and question generation. Experiments on the OR-ShARC dataset show the effectiveness of our method, which achieves new state-of-the-art results.
  arXiv  Detail & Related papers  (2021-08-28T08:04:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.