Bridging the Stability-Expressivity Gap: Synthetic Data Scaling and Preference Alignment for Low-Resource Spoken Language Models

• URL: http://arxiv.org/abs/2605.27383v1
• Date: Fri, 10 Apr 2026 12:44:27 GMT
• Title: Bridging the Stability-Expressivity Gap: Synthetic Data Scaling and Preference Alignment for Low-Resource Spoken Language Models
• Abstract summary: Spoken Language Models (SLMs) have emerged as a promising paradigm for speech synthesis by bypassing explicit grapheme-to-phoneme pipelines.<n>In practice, synthetic data has become the primary strategy for scaling SLMs in such settings.<n>We show that this reliance introduces a fundamental trade-off, which we term the Stability-Expressivity Gap.
• Score: 11.897185319653744
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Spoken Language Models (SLMs) have emerged as a promising paradigm for speech synthesis by bypassing explicit grapheme-to-phoneme pipelines. However, their effectiveness in low-resource languages remains fundamentally limited by the scarcity of transcribed speech. In practice, synthetic data has become the primary strategy for scaling SLMs in such settings, providing reliable phonetic supervision when real data is insufficient. In this work, we show that this reliance introduces a fundamental trade-off, which we term the Stability-Expressivity Gap: while synthetic data improves phonetic accuracy, it progressively suppresses prosodic variability, ultimately leading to a collapse of expressivity (Synthetic Erosion). To bridge this gap, we propose two self-alignment frameworks. Disentanglement-Guided Self-Alignment (DGSA) recovers expressivity for complex languages by exploiting prosody-timbre separation. For regimes where authentic references are exceptionally limited, Temperature-Driven Self-Critique (TDSC) stabilizes generation through automated exploration and filtering. Our approach outperforms strong commercial systems, including ElevenLabs and Gemini Pro, and enables the first zero-shot voice cloning capability for Lao.

Related papers

• Provable Sparse Inversion and Token Relabel Enhanced One-shot Federated Learning with ViTs [77.42033827176806]
  One-Shot Federated Learning, where a central server learns a global model in a single communication round, has emerged as a promising paradigm.<n>We propose a novel Federated Model Inversion and Token Relabel framework, which trains the global model by fully exploiting all patches of synthetic images.
  arXiv  Detail & Related papers  (2026-05-11T15:49:08Z)
• ZeSTA: Zero-Shot TTS Augmentation with Domain-Conditioned Training for Data-Efficient Personalized Speech Synthesis [3.1848820580333737]
  We investigate the use of zero-shot text-to-speech (ZS-TTS) as a data augmentation source for low-resource personalized speech synthesis.<n>We propose ZeSTA, a simple domain-conditioned training framework that distinguishes real and synthetic speech.
  arXiv  Detail & Related papers  (2026-03-04T16:04:02Z)
• DiTSinger: Scaling Singing Voice Synthesis with Diffusion Transformer and Implicit Alignment [13.149605745750245]
  We introduce a two-stage pipeline: a compact seed set of human-sung recordings is constructed by pairing fixed melodies with diverse lyrics, and melody-specific models are trained to synthesize over 500 hours of Chinese singing data.<n>We propose DiTSinger, a Diffusion Transformer with RoPE and qk-norm, systematically scaled in depth, width, and resolution for enhanced fidelity.
  arXiv  Detail & Related papers  (2025-10-10T05:39:45Z)
• StableToken: A Noise-Robust Semantic Speech Tokenizer for Resilient SpeechLLMs [54.229363096087866]
  Speech tokenizers are not robust to meaning-irrelevant acoustic perturbations.<n>This instability stems from two flaws: a brittle single-path quantization architecture and a distant training signal.<n>We introduce StableToken, a tokenizer that achieves stability through a consensus-driven mechanism.
  arXiv  Detail & Related papers  (2025-09-26T11:32:51Z)
• DRES: Benchmarking LLMs for Disfluency Removal [27.083825614818135]
  Disfluencies, such as "um," "uh," interjections, parentheticals, and edited statements, remain a persistent challenge for speech-driven systems.<n>Disfluency Removal Evaluation Suite, a controlled text-level benchmark, establishes a reproducible semantic upper bound for this task.<n>DRES builds on human-annotated Switchboard transcripts, isolating disfluency removal from ASR errors and acoustic variability.
  arXiv  Detail & Related papers  (2025-09-24T17:08:12Z)
• MegaTTS 3: Sparse Alignment Enhanced Latent Diffusion Transformer for Zero-Shot Speech Synthesis [56.25862714128288]
  This paper introduces textitMegaTTS 3, a zero-shot text-to-speech (TTS) system featuring an innovative sparse alignment algorithm.<n>Specifically, we provide sparse alignment boundaries to MegaTTS 3 to reduce the difficulty of alignment without limiting the search space.<n>Experiments demonstrate that MegaTTS 3 achieves state-of-the-art zero-shot TTS speech quality and supports highly flexible control over accent intensity.
  arXiv  Detail & Related papers  (2025-02-26T08:22:00Z)
• Refining Sentence Embedding Model through Ranking Sentences Generation with Large Language Models [60.00178316095646]
  Sentence embedding is essential for many NLP tasks, with contrastive learning methods achieving strong performance using datasets like NLI.<n>Recent studies leverage large language models (LLMs) to generate sentence pairs, reducing annotation dependency.<n>We propose a method for controlling the generation direction of LLMs in the latent space. Unlike unconstrained generation, the controlled approach ensures meaningful semantic divergence.<n> Experiments on multiple benchmarks demonstrate that our method achieves new SOTA performance with a modest cost in ranking sentence synthesis.
  arXiv  Detail & Related papers  (2025-02-19T12:07:53Z)
• It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognition [70.77292069313154]
  Large language models (LLMs) can be successfully used for generative error correction (GER) on top of the automatic speech recognition (ASR) output. In this work, we aim to overcome such a limitation by infusing acoustic information before generating the predicted transcription through a novel late fusion solution termed Uncertainty-Aware Dynamic Fusion (UADF)
  arXiv  Detail & Related papers  (2024-02-08T07:21:45Z)
• HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models [81.56455625624041]
  We introduce the first open-source benchmark to utilize external large language models (LLMs) for ASR error correction. The proposed benchmark contains a novel dataset, HyPoradise (HP), encompassing more than 334,000 pairs of N-best hypotheses. LLMs with reasonable prompt and its generative capability can even correct those tokens that are missing in N-best list.
  arXiv  Detail & Related papers  (2023-09-27T14:44:10Z)
• Steering Language Generation: Harnessing Contrastive Expert Guidance and Negative Prompting for Coherent and Diverse Synthetic Data Generation [0.0]
  Large Language Models (LLMs) hold immense potential to generate synthetic data of high quality and utility. We introduce contrastive expert guidance, where the difference between the logit distributions of fine-tuned and base language models is emphasised. We deem this dual-pronged approach to logit reshaping as STEER: Semantic Text Enhancement via Embedding Repositioning.
  arXiv  Detail & Related papers  (2023-08-15T08:49:14Z)
• EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesis [49.04496602282718]
  We introduce Expresso, a high-quality expressive speech dataset for textless speech synthesis. This dataset includes both read speech and improvised dialogues rendered in 26 spontaneous expressive styles. We evaluate resynthesis quality with automatic metrics for different self-supervised discrete encoders.
  arXiv  Detail & Related papers  (2023-08-10T17:41:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.