Unlocking Fine-Grained and Within-Utterance Speaking Style Control in Prompt-Based Text-to-Speech Models

• URL: http://arxiv.org/abs/2605.27376v1
• Date: Thu, 09 Apr 2026 01:06:26 GMT
• Title: Unlocking Fine-Grained and Within-Utterance Speaking Style Control in Prompt-Based Text-to-Speech Models
• Abstract summary: We propose techniques to achieve continuous style attribute across utterances and time-varying style transitions within a single utterance.<n>Our proposed inter-utterance achieves a 99-100% success rate in gender conversion, up to 36 Hz pitch variation, and up to 1.6 syllables-per-second speed change.
• Score: 11.115821694268716
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: While prompt-based text-to-speech (TTS) models enable natural language-driven speaking style control, they often provide limited fine-grained control and apply a single global style across an utterance. This restricts practical use cases that require continuous style attribute interpolation across utterances and time-varying style transitions within a single utterance. In this paper, we propose novel techniques to achieve both capabilities in existing prompt-based TTS models. For inter-utterance style interpolation, we compute direction vectors between contrastive style prompts in the embedding space and perform simple interpolation, enabling smooth transitions between style characteristics. For intra-utterance style transition, we first identify a strong attention bias toward early tokens in autoregressive TTS decoders, causing the initial audio realization to dominate subsequent generation. To mitigate this effect, we introduce KV-cache swapping and sliding-window attention masking. Experiments demonstrate that our proposed inter-utterance interpolation achieves a 99-100% success rate in gender conversion, up to 36 Hz pitch variation, and up to 1.6 syllables-per-second speed change. Our intra-utterance transition maintains a speaker similarity of 0.81-0.91 and achieves perceptual smoothness scores of 3.48-4.48.

Related papers

• How Do Instructions Shape Speech? Cross-Attention Attribution for Style-Captioned Text-to-Speech [8.950816086845114]
  Style-captioned text-to-speech systems use natural language to control voice characteristics, but how individual words influence acoustic output remains unclear.<n>We propose cross-attention attribution for speech diffusion models, adapting the DAAM framework to the speech domain for the first time.<n>This is the first study of how natural language influences cross-attention in speech diffusion models.
  arXiv  Detail & Related papers  (2026-06-18T17:47:32Z)
• HiStyle: Hierarchical Style Embedding Predictor for Text-Prompt-Guided Controllable Speech Synthesis [17.743822016045446]
  Controllable speech synthesis refers to the precise control of speaking style by manipulating specific prosodic and paralinguistic attributes.<n>We propose HiStyle, a two-stage style embedding predictor that hierarchically predicts style embeddings conditioned on textual prompts.
  arXiv  Detail & Related papers  (2025-09-30T06:31:12Z)
• ReverBERT: A State Space Model for Efficient Text-Driven Speech Style Transfer [0.0]
  We present emphReverBERT, an efficient framework for text-driven speech style transfer.<n>Unlike image domain techniques, our method operates in the speech space and integrates a discrete Fourier transform of latent speech features.<n>Experiments on benchmark speech corpora demonstrate that emphReverBERT significantly outperforms baselines in terms of naturalness, expressiveness, and computational efficiency.
  arXiv  Detail & Related papers  (2025-03-26T21:11:17Z)
• MegaTTS 3: Sparse Alignment Enhanced Latent Diffusion Transformer for Zero-Shot Speech Synthesis [56.25862714128288]
  This paper introduces textitMegaTTS 3, a zero-shot text-to-speech (TTS) system featuring an innovative sparse alignment algorithm.<n>Specifically, we provide sparse alignment boundaries to MegaTTS 3 to reduce the difficulty of alignment without limiting the search space.<n>Experiments demonstrate that MegaTTS 3 achieves state-of-the-art zero-shot TTS speech quality and supports highly flexible control over accent intensity.
  arXiv  Detail & Related papers  (2025-02-26T08:22:00Z)
• DPI-TTS: Directional Patch Interaction for Fast-Converging and Style Temporal Modeling in Text-to-Speech [43.45691362372739]
  We propose a method called Directional Patch Interaction for Text-to-Speech (DPI-TTS) DPI-TTS employs a low-to-high frequency, frame-by-frame progressive inference approach that aligns more closely with acoustic properties. Experimental results demonstrate that our method increases the training speed by nearly 2 times and significantly outperforms the baseline models.
  arXiv  Detail & Related papers  (2024-09-18T09:36:55Z)
• SimpleSpeech 2: Towards Simple and Efficient Text-to-Speech with Flow-based Scalar Latent Transformer Diffusion Models [64.40250409933752]
  We build upon our previous publication by implementing a simple and efficient non-autoregressive (NAR) TTS framework, termed SimpleSpeech 2. SimpleSpeech 2 effectively combines the strengths of both autoregressive (AR) and non-autoregressive (NAR) methods. We show a significant improvement in generation performance and generation speed compared to our previous work and other state-of-the-art (SOTA) large-scale TTS models.
  arXiv  Detail & Related papers  (2024-08-25T17:07:39Z)
• ControlSpeech: Towards Simultaneous and Independent Zero-shot Speaker Cloning and Zero-shot Language Style Control [50.27383290553548]
  ControlSpeech is a text-to-speech (TTS) system capable of fully cloning the speaker's voice and enabling arbitrary control and adjustment of speaking style.<n>We show that ControlSpeech exhibits comparable or state-of-the-art (SOTA) performance in terms of controllability, timbre similarity, audio quality, robustness, and generalizability.
  arXiv  Detail & Related papers  (2024-06-03T11:15:16Z)
• Pruning Self-Attention for Zero-Shot Multi-Speaker Text-to-Speech [26.533600745910437]
  We propose an effective pruning method for a transformer known as sparse attention, to improve the TTS model's generalization abilities. We also propose a new differentiable pruning method that allows the model to automatically learn the thresholds.
  arXiv  Detail & Related papers  (2023-08-28T21:25:05Z)
• GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speech Synthesis [68.42632589736881]
  This paper proposes GenerSpeech, a text-to-speech model towards high-fidelity zero-shot style transfer of OOD custom voice. GenerSpeech decomposes the speech variation into the style-agnostic and style-specific parts by introducing two components. Our evaluations on zero-shot style transfer demonstrate that GenerSpeech surpasses the state-of-the-art models in terms of audio quality and style similarity.
  arXiv  Detail & Related papers  (2022-05-15T08:16:02Z)
• Fine-grained style control in Transformer-based Text-to-speech Synthesis [78.92428622630861]
  We present a novel architecture to realize fine-grained style control on the Transformer-based text-to-speech synthesis (TransformerTTS) We model the speaking style by extracting a time sequence of local style tokens (LST) from the reference speech. Experiments show that with fine-grained style control, our system performs better in terms of naturalness, intelligibility, and style transferability.
  arXiv  Detail & Related papers  (2021-10-12T19:50:02Z)
• Meta-StyleSpeech : Multi-Speaker Adaptive Text-to-Speech Generation [63.561944239071615]
  StyleSpeech is a new TTS model which synthesizes high-quality speech and adapts to new speakers. With SALN, our model effectively synthesizes speech in the style of the target speaker even from single speech audio. We extend it to Meta-StyleSpeech by introducing two discriminators trained with style prototypes, and performing episodic training.
  arXiv  Detail & Related papers  (2021-06-06T15:34:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.