Scaling Properties of Continuous Diffusion Spoken Language Models

• URL: http://arxiv.org/abs/2604.24416v1
• Date: Mon, 27 Apr 2026 12:45:18 GMT
• Title: Scaling Properties of Continuous Diffusion Spoken Language Models
• Abstract summary: Speech-only spoken language models (SLMs) lag behind text and text-speech models in performance.<n>Recent discrete autoregressive (AR) SLMs indicate significant computational and data demands to match text models.<n>To quantify the SLMs linguistic quality, we introduce the phoneme Jensen-Shannon divergence (pJSD) metric.
• Score: 25.864710231019483
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Speech-only spoken language models (SLMs) lag behind text and text-speech models in performance, with recent discrete autoregressive (AR) SLMs indicating significant computational and data demands to match text models. Since discretizing continuous speech for AR creates bottlenecks, we explore whether continuous diffusion (CD) SLM is more viable. To quantify the SLMs linguistic quality, we introduce the phoneme Jensen-Shannon divergence (pJSD) metric. Our analysis reveals CD SLMs, mirroring AR behavior, exhibit scaling laws for validation loss and pJSD, and show optimal token-to-parameter ratios decreasing as compute scales. However, for the latter, loss becomes insensitive to choice of data and model sizes, showing potential for fast inference. Scaling CD SLMs to 16B parameters with tens of millions of hours of conversational data enables generation of emotive, prosodic, multi-speaker, multilingual speech, though achieving long-form coherence remains a significant challenge.

Related papers

• On the Effect of Segmentation Width and Cluster Size on Speech Resynthesis and Continuation in Generative Spoken Language Models [25.93282682937446]
  Generative Spoken Language Modeling (GSLM) enables text-free speech modeling by training language models.<n>We investigate the performance of GSLM on speech synthesis and continuation using discrete speech representations.
  arXiv  Detail & Related papers  (2026-06-22T12:58:26Z)
• Adapting Text LLMs to Speech via Multimodal Depth Up-Scaling [52.02344262645619]
  We propose Multimodal Depth Upscaling, where new transformer layers are inserted into a frozen text LLM and only the added layers are trained on speech data.<n>Experiments with SmolLM2-360M and SmolLM2-1.7B on 48k hours of English Automatic Speech Recognition (ASR) data show that depth up-scaling achieves ASR comparable to full fine-tuning.<n>We further show that incorporating E-Branchformer, an architecture designed for speech recognition, as the inserted layers achieves ASR that matches or surpasses full fine-tuning on the larger model while reducing text degradation by over 75% with 60% fewer trainable parameters
  arXiv  Detail & Related papers  (2026-04-01T05:16:06Z)
• SLM-SS: Speech Language Model for Generative Speech Separation [47.06391017558454]
  We propose SLM-SS, a novel approach that applies speech language models to speech separation.<n>We frame SS as discrete multi-codebook sequence generation, usingcoder models to map quantized speech mixtures to target tokens.<n>Our approach shows significantly better preservation of speech intelligibility, leading to improved linguistic consistency in a variety of downstream tasks.
  arXiv  Detail & Related papers  (2026-01-27T12:22:43Z)
• Reproducing and Dissecting Denoising Language Models for Speech Recognition [31.91567892562116]
  Denoising language models (DLMs) have been proposed as a powerful alternative to traditional language models (LMs) for automatic speech recognition (ASR)<n>This paper presents the first independent, large-scale empirical study of DLMs.
  arXiv  Detail & Related papers  (2025-12-15T17:33:22Z)
• Scaling Behavior of Discrete Diffusion Language Models [74.72926629897636]
  We study the scaling behavior of discrete diffusion language models (DLMs) on different noise types.<n>Our experiments reveal that the scaling behavior of DLMs strongly depends on the noise type and is considerably different from ALMs.<n>We scale our uniform diffusion model up to 10B parameters trained for $1022$ FLOPs, confirming the predicted scaling behavior and making it the largest publicly known uniform diffusion model to date.
  arXiv  Detail & Related papers  (2025-12-11T17:54:10Z)
• Scaling Analysis of Interleaved Speech-Text Language Models [22.61336359340435]
  Existing Speech Language Model (SLM) scaling analysis predicts that SLMs require much more compute and data compared to text.<n>Modern SLMs are often initialised from pre-trained TextLMs using speech-text interleaving to allow knowledge transfer.<n>We conduct scaling analysis of interleaved SLMs by training several dozen and analysing the scaling trends.<n>Results suggest that our scaled up model achieves comparable semantic speech performance to leading models, while using less compute and data.
  arXiv  Detail & Related papers  (2025-04-03T08:46:56Z)
• VoiceTextBlender: Augmenting Large Language Models with Speech Capabilities via Single-Stage Joint Speech-Text Supervised Fine-Tuning [64.56272011710735]
  We propose a novel single-stage joint speech-text SFT approach on the low-rank adaptation (LoRA) of the large language models (LLMs) backbone.<n>Compared to previous SpeechLMs with 7B or 13B parameters, our 3B model demonstrates superior performance across various speech benchmarks.
  arXiv  Detail & Related papers  (2024-10-23T00:36:06Z)
• SimpleSpeech 2: Towards Simple and Efficient Text-to-Speech with Flow-based Scalar Latent Transformer Diffusion Models [64.40250409933752]
  We build upon our previous publication by implementing a simple and efficient non-autoregressive (NAR) TTS framework, termed SimpleSpeech 2. SimpleSpeech 2 effectively combines the strengths of both autoregressive (AR) and non-autoregressive (NAR) methods. We show a significant improvement in generation performance and generation speed compared to our previous work and other state-of-the-art (SOTA) large-scale TTS models.
  arXiv  Detail & Related papers  (2024-08-25T17:07:39Z)
• Minimally-Supervised Speech Synthesis with Conditional Diffusion Model and Language Model: A Comparative Study of Semantic Coding [57.42429912884543]
  We propose Diff-LM-Speech, Tetra-Diff-Speech and Tri-Diff-Speech to solve high dimensionality and waveform distortion problems. We also introduce a prompt encoder structure based on a variational autoencoder and a prosody bottleneck to improve prompt representation ability. Experimental results show that our proposed methods outperform baseline methods.
  arXiv  Detail & Related papers  (2023-07-28T11:20:23Z)
• A Cheaper and Better Diffusion Language Model with Soft-Masked Noise [62.719656543880596]
  Masked-Diffuse LM is a novel diffusion model for language modeling, inspired by linguistic features in languages. Specifically, we design a linguistic-informed forward process which adds corruptions to the text through strategically soft-masking to better noise the textual data. We demonstrate that our Masked-Diffuse LM can achieve better generation quality than the state-of-the-art diffusion models with better efficiency.
  arXiv  Detail & Related papers  (2023-04-10T17:58:42Z)
• Towards Language Modelling in the Speech Domain Using Sub-word Linguistic Units [56.52704348773307]
  We propose a novel LSTM-based generative speech LM based on linguistic units including syllables and phonemes. With a limited dataset, orders of magnitude smaller than that required by contemporary generative models, our model closely approximates babbling speech. We show the effect of training with auxiliary text LMs, multitask learning objectives, and auxiliary articulatory features.
  arXiv  Detail & Related papers  (2021-10-31T22:48:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.