GLASS: GRPO-Trained LoRA for Acoustic Style Steering in Zero-Shot Text-to-Speech

• URL: http://arxiv.org/abs/2606.05889v1
• Date: Thu, 04 Jun 2026 08:58:57 GMT
• Title: GLASS: GRPO-Trained LoRA for Acoustic Style Steering in Zero-Shot Text-to-Speech
• Abstract summary: We propose GLASS, a framework for composable acoustic style control in zero-shot autoregressive text-to-speech (TTS)<n>In TTS, a speaker prompt often entangles speaker identity with prosodic attributes such as speaking rate and pitch, making it difficult to change style without changing the prompt itself.<n>Experiments on speaking rate and pitch control show targeted style shifts while preserving naturalness, speaker similarity, and intelligibility, and demonstrate smooth and multi-axis composition across independently trained adapters.
• Score: 12.143236645802787
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: We propose GLASS, a framework for composable acoustic style control in zero-shot autoregressive text-to-speech (TTS) that learns controls from post-generation rewards rather than style labels. In zero-shot TTS, a speaker prompt often entangles speaker identity with prosodic attributes such as speaking rate and pitch, making it difficult to change style without changing the prompt itself. GLASS instead treats each acoustic attribute as a reward-defined control direction. For each control axis, GLASS freezes the TTS backbone and trains one lightweight LoRA adapter with Group Relative Policy Optimization (GRPO), using speech-token length and mean F0 as style rewards and WER as an intelligibility anchor. Because each control is represented as a LoRA weight update, independently trained adapters can be swapped, interpolated, and composed through linear LoRA arithmetic without retraining the backbone. Experiments on speaking rate and pitch control show targeted style shifts while preserving naturalness, speaker similarity, and intelligibility, and demonstrate smooth interpolation and multi-axis composition across independently trained adapters.

Related papers

• Unlocking Fine-Grained and Within-Utterance Speaking Style Control in Prompt-Based Text-to-Speech Models [11.115821694268716]
  We propose techniques to achieve continuous style attribute across utterances and time-varying style transitions within a single utterance.<n>Our proposed inter-utterance achieves a 99-100% success rate in gender conversion, up to 36 Hz pitch variation, and up to 1.6 syllables-per-second speed change.
  arXiv  Detail & Related papers  (2026-04-09T01:06:26Z)
• ID-LoRA: Identity-Driven Audio-Video Personalization with In-Context LoRA [63.95311560556552]
  Existing video personalization methods preserve visual likeness but treat video and audio separately.<n>We propose ID-LoRA, which jointly generates a subject's appearance and voice in a single model.<n>In human preference studies, ID-LoRA is preferred over Kling 2.6 Pro by 73% of annotators for voice similarity and 65% for speaking style.
  arXiv  Detail & Related papers  (2026-03-10T22:23:36Z)
• Lombard Speech Synthesis for Any Voice with Controllable Style Embeddings [61.29502937013759]
  controllable text-to-speech (TTS) system capable of synthesizing Lombard speech for any speaker without requiring explicit Lombard data during training.<n>Our approach leverages style embeddings learned from a large, prosodically diverse dataset and analyzes their correlation with Lombard attributes using principal component analysis (PCA).
  arXiv  Detail & Related papers  (2026-01-19T11:25:19Z)
• ReStyle-TTS: Relative and Continuous Style Control for Zero-Shot Speech Synthesis [35.41874154907003]
  Zero-shot text-to-speech models can clone a speaker's timbre from a short reference audio, but they also strongly inherit the speaking style present in the reference.<n>We propose ReStyle-TTS, a framework that enables continuous and reference-relative style control in zero-shot TTS.
  arXiv  Detail & Related papers  (2026-01-07T06:23:23Z)
• HiStyle: Hierarchical Style Embedding Predictor for Text-Prompt-Guided Controllable Speech Synthesis [17.743822016045446]
  Controllable speech synthesis refers to the precise control of speaking style by manipulating specific prosodic and paralinguistic attributes.<n>We propose HiStyle, a two-stage style embedding predictor that hierarchically predicts style embeddings conditioned on textual prompts.
  arXiv  Detail & Related papers  (2025-09-30T06:31:12Z)
• VStyle: A Benchmark for Voice Style Adaptation with Spoken Instructions [66.93932684284695]
  Spoken language models (SLMs) have emerged as a unified paradigm for speech understanding and generation.<n>We introduce Voice Style Adaptation (VSA), a new task that examines whether SLMs can modify their speaking style.<n>We present VStyle, a benchmark covering four categories of speech generation: acoustic attributes, natural language instruction, role play, and implicit empathy.<n>We also introduce the Large Audio Language Model as a Judge (LALM as a Judge) framework, which progressively evaluates outputs along textual faithfulness, style adherence, and naturalness.
  arXiv  Detail & Related papers  (2025-09-09T14:28:58Z)
• InstructTTSEval: Benchmarking Complex Natural-Language Instruction Following in Text-to-Speech Systems [48.42417538526542]
  Text-to-Speech systems rely on fixed style labels or inserting a speech prompt to control these cues.<n>Recent attempts seek to employ natural-language instructions to modulate paralinguistic features.<n>InstructTTSEval is a benchmark for measuring the capability of complex natural-language style control.
  arXiv  Detail & Related papers  (2025-06-19T15:08:01Z)
• StyleSpeech: Parameter-efficient Fine Tuning for Pre-trained Controllable Text-to-Speech [13.713209707407712]
  StyleSpeech is a novel Text-to-Speech(TTS) system that enhances the naturalness and accuracy of synthesized speech. Building upon existing TTS technologies, StyleSpeech incorporates a unique Style Decorator structure that enables deep learning models to simultaneously learn style and phoneme features. LoRA allows efficient adaptation of style features in pre-trained models.
  arXiv  Detail & Related papers  (2024-08-27T00:37:07Z)
• ControlSpeech: Towards Simultaneous and Independent Zero-shot Speaker Cloning and Zero-shot Language Style Control [50.27383290553548]
  ControlSpeech is a text-to-speech (TTS) system capable of fully cloning the speaker's voice and enabling arbitrary control and adjustment of speaking style.<n>We show that ControlSpeech exhibits comparable or state-of-the-art (SOTA) performance in terms of controllability, timbre similarity, audio quality, robustness, and generalizability.
  arXiv  Detail & Related papers  (2024-06-03T11:15:16Z)
• Controllable speech synthesis by learning discrete phoneme-level prosodic representations [53.926969174260705]
  We present a novel method for phoneme-level prosody control of F0 and duration using intuitive discrete labels. We propose an unsupervised prosodic clustering process which is used to discretize phoneme-level F0 and duration features from a multispeaker speech dataset.
  arXiv  Detail & Related papers  (2022-11-29T15:43:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.