Thinking-while-speaking: A Controlled, Interleaved Reasoning Method for Real-Time Speech Generation

• URL: http://arxiv.org/abs/2605.20946v1
• Date: Wed, 20 May 2026 09:32:35 GMT
• Title: Thinking-while-speaking: A Controlled, Interleaved Reasoning Method for Real-Time Speech Generation
• Abstract summary: Key challenge is maintaining fluent speech while performing deep reasoning.<n>Our method, InterRS, tackles this by inserting reasoning steps only during natural speech generation.<n>We introduce a novel pipeline to generate seamlessly interleaved audio data.
• Score: 15.297424620191158
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The thinking-while-speaking paradigm aims to make AI communication more human. A key challenge is maintaining fluent speech while performing deep reasoning. Our method, InterRS, tackles this by inserting reasoning steps only during natural speech generation. This requires high-quality data where reasoning and speech are precisely aligned, and the length ratio are under controlled. We introduce a novel pipeline to generate such seamlessly interleaved audio data. To train our model, we combine interleaved SFT with refined data and reinforcement learning with two new rewards: a TA-Balance Reward to manage timing and thinking-answer ratio, and a Linguistic Quality Reward to refine expression. Experiments show our approach achieves 13% better performance on mathmatical and logic benchmarks while generating instant response like a spoken-language instruct model which outputs fast CoT response. Furthermore, our method generates more natural and fluent answers than prior methods.

Related papers

• DiffuSpeech: Silent Thought, Spoken Answer via Unified Speech-Text Diffusion [23.01044837428522]
  We introduce textbfSilent Thought, Spoken Answer'' -- a paradigm where speech LLMs generate internal text reasoning alongside spoken responses.<n>We present method, the first diffusion-based speech-text language model supporting both understanding and generation.<n>Experiments show method achieves state-of-the-art speech-to-speech QA accuracy, outperforming the best baseline by up to 9 points.
  arXiv  Detail & Related papers  (2026-01-30T12:08:33Z)
• MOSS-Speech: Towards True Speech-to-Speech Models Without Text Guidance [66.74042564585942]
  MOSS-Speech is a true speech-to-speech large language model that directly understands and generates speech without relying on text guidance.<n>Our work establishes a new paradigm for expressive and efficient end-to-end speech interaction.
  arXiv  Detail & Related papers  (2025-10-01T04:32:37Z)
• Think, Verbalize, then Speak: Bridging Complex Thoughts and Comprehensible Speech [41.625380059502675]
  Think-Verbalize-Speak is a framework that decouples reasoning from spoken delivery.<n>We also introduce ReVerT, a latency-efficient verbalizer based on incremental and asynchronous summarization.<n> Experiments across multiple benchmarks show that our method enhances speech naturalness and conciseness with minimal impact on reasoning.
  arXiv  Detail & Related papers  (2025-09-19T14:34:22Z)
• Mini-Omni-Reasoner: Token-Level Thinking-in-Speaking in Large Speech Models [80.75260664100644]
  Mini-Omni-Reasoner is a framework that enables reasoning within speech via a novel "Thinking-in-Speaking" formulation.<n>It interleaves silent reasoning tokens with spoken response tokens at the token level.<n>It achieves a +19.1% gain in arithmetic reasoning and +6.4% in contextual understanding, with shorter outputs and zero decoding latency.
  arXiv  Detail & Related papers  (2025-08-18T15:14:04Z)
• Towards Efficient Speech-Text Jointly Decoding within One Speech Language Model [76.06585781346601]
  Speech language models (Speech LMs) enable end-to-end speech-text modelling within a single model.<n>The choice of speech-text jointly decoding paradigm plays a critical role in performance, efficiency, and alignment quality.
  arXiv  Detail & Related papers  (2025-06-04T23:53:49Z)
• Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning [53.57895922042783]
  Large Language Models (LLMs) excel at reasoning and planning when trained on chainof-thought (CoT) data.<n>We propose a hybrid representation of the reasoning process, where we partially abstract away the initial reasoning steps using latent discrete tokens.
  arXiv  Detail & Related papers  (2025-02-05T15:33:00Z)
• Predictive Speech Recognition and End-of-Utterance Detection Towards Spoken Dialog Systems [55.99999020778169]
  We study a function that can predict the forthcoming words and estimate the time remaining until the end of an utterance. We develop a cross-attention-based algorithm that incorporates both acoustic and linguistic information. Results demonstrate the proposed model's ability to predict upcoming words and estimate future EOU events up to 300ms prior to the actual EOU.
  arXiv  Detail & Related papers  (2024-09-30T06:29:58Z)
• Learning to Ask Conversational Questions by Optimizing Levenshtein Distance [83.53855889592734]
  We introduce a Reinforcement Iterative Sequence Editing (RISE) framework that optimize the minimum Levenshtein distance (MLD) through explicit editing actions. RISE is able to pay attention to tokens that are related to conversational characteristics. Experimental results on two benchmark datasets show that RISE significantly outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-06-30T08:44:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.