Ready to Speak: Aligning LLMs for TTS-Friendly Text Generation

• URL: http://arxiv.org/abs/2609.01246v1
• Date: Tue, 01 Sep 2026 13:44:43 GMT
• Title: Ready to Speak: Aligning LLMs for TTS-Friendly Text Generation
• Abstract summary: Current Large Language Models (LLMs) are primarily optimized for written text, often producing outputs that are grammatically correct and helpful yet poorly suited for spoken delivery via Text-to-Speech (TTS)<n>In this work, we study how to make LLMs generate TTS-friendly text, which we frame as a preference alignment problem.<n>We introduce two preference datasets spanning different target domains, CORA and Recipe, which contain paired TTS-friendly and TTS-unfriendly responses.
• Score: 16.8983983115275
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Current Large Language Models (LLMs) are primarily optimized for written text, often producing outputs that are grammatically correct and helpful yet poorly suited for spoken delivery via Text-to-Speech (TTS). In this work, we study how to make LLMs natively generate TTS-friendly text, which we frame as a preference alignment problem: instead of relying on downstream rewriting modules, we directly align LLMs to generate text optimized for spoken delivery. We introduce two preference datasets spanning different target domains, CORA and Recipe, which contain paired TTS-friendly and TTS-unfriendly responses. We further propose an evaluation suite combining a pattern-based heuristic metric, a TTS$\to$ASR evaluation pipeline, and a MUSHRA listening study with human judges. Our experiments compare the recently proposed Feature-aware Sampling and Tuning (FaST) framework -- leveraging interpretable features instead of a black-box reward model -- against an array of alignment baselines on the TTS-friendly generation task. Notably, we found that FaST achieves the best overall tradeoff between TTS-friendliness and helpfulness across various settings. We also identified a strong correlation between our different metrics, highlighting the ability to reliably assess TTS-friendliness via an efficient heuristic.

Related papers

• Revisiting Direct Speech-to-Text Translation with Speech LLMs: Better Scaling than CoT Prompting? [13.202203902821333]
  We systematically compare Chain-of-Thought (CoT) and Direct prompting under increasing amounts of Speech-to-Text Translation (S2TT) data.<n>Our results show that Direct improves more consistently as the amount of data increases, suggesting that it may become a more effective approach as larger S2TT resources are created.
  arXiv  Detail & Related papers  (2025-10-03T15:23:32Z)
• EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic, Expressiveness, and Linguistic Challenges Using Model-as-a-Judge [25.51206687438354]
  We introduce $textitEmergentTTS-Eval, a comprehensive benchmark covering six challenging TTS scenarios.<n>Our framework automates both test-case generation and evaluation, making the benchmark easily accessible.<n>We evaluate state-of-the-art open-source and proprietary TTS systems, such as 11Labs, Deepgram, and OpenAI's 4o-mini-TTS, on EmergentTTS-Eval.
  arXiv  Detail & Related papers  (2025-05-29T02:36:24Z)
• TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modeling [46.60911294356232]
  We introduce Text-Aligned Speech Tokenization and Embedding (TASTE) to align speech token with corresponding text transcription during the tokenization stage.<n>We conduct extensive experiments and show that TASTE can preserve essential paralinguistic information while dramatically reducing the token sequence length.<n> Experimental results show that TASTE-based SLMs perform comparable to previous work on SALMON and StoryCloze.
  arXiv  Detail & Related papers  (2025-04-09T17:14:33Z)
• Evaluating Text Style Transfer Evaluation: Are There Any Reliable Metrics? [9.234136424254261]
  Text style transfer (TST) is the task of transforming a text to reflect a particular style while preserving its original content.<n>Using human evaluation is ideal but costly, as is common in other natural language processing (NLP) tasks.<n>In this paper, we examine both set of existing and novel metrics from broader NLP tasks for TST evaluation.
  arXiv  Detail & Related papers  (2025-02-07T07:39:17Z)
• Preference Alignment Improves Language Model-Based TTS [76.70693823683091]
  preference alignment algorithms adjust LMs to align with the preferences of reward models, enhancing the desirability of the generated content. With a 1.15B parameter LM-based TTS model, we demonstrate that preference alignment consistently improves intelligibility, speaker similarity, and proxy subjective evaluation scores.
  arXiv  Detail & Related papers  (2024-09-19T01:58:19Z)
• Blending LLMs into Cascaded Speech Translation: KIT's Offline Speech Translation System for IWSLT 2024 [61.189875635090225]
  Large Language Models (LLMs) are currently under exploration for various tasks, including Automatic Speech Recognition (ASR), Machine Translation (MT), and even End-to-End Speech Translation (ST)
  arXiv  Detail & Related papers  (2024-06-24T16:38:17Z)
• Are Large Language Models Actually Good at Text Style Transfer? [0.17751300245073598]
  We analyze the performance of large language models (LLMs) on Text Style Transfer (TST) TST involves modifying the linguistic style of a text while preserving its core content. We evaluate the capabilities of pre-trained LLMs using zero-shot and few-shot prompting as well as parameter-efficient finetuning on publicly available datasets.
  arXiv  Detail & Related papers  (2024-06-09T18:45:41Z)
• Prefix Text as a Yarn: Eliciting Non-English Alignment in Foundation Language Model [50.339632513018934]
  supervised fine-tuning (SFT) has been a straightforward approach for tailoring the output of foundation large language model (LLM) to specific preferences. We critically examine this hypothesis within the scope of cross-lingual generation tasks. We introduce a novel training-free alignment method named PreTTY, which employs minimal task-related prior tokens.
  arXiv  Detail & Related papers  (2024-04-25T17:19:36Z)
• Speech Translation with Large Language Models: An Industrial Practice [64.5419534101104]
  We introduce LLM-ST, a novel and effective speech translation model constructed upon a pre-trained large language model (LLM) By integrating the large language model (LLM) with a speech encoder and employing multi-task instruction tuning, LLM-ST can produce accurate timestamped transcriptions and translations. Through rigorous experimentation on English and Chinese datasets, we showcase the exceptional performance of LLM-ST.
  arXiv  Detail & Related papers  (2023-12-21T05:32:49Z)
• Code-Mixed Text to Speech Synthesis under Low-Resource Constraints [6.544954579068865]
  We describe our approaches for production quality code-mixed Hindi-English TTS systems built for e-commerce applications. We propose a data-oriented approach by utilizing monolingual data sets in individual languages. We show that such single script bi-lingual training without any code-mixing works well for pure code-mixed test sets.
  arXiv  Detail & Related papers  (2023-12-02T10:40:38Z)
• ESPnet2-TTS: Extending the Edge of TTS Research [62.92178873052468]
  ESPnet2-TTS is an end-to-end text-to-speech (E2E-TTS) toolkit. New features include: on-the-fly flexible pre-processing, joint training with neural vocoders, and state-of-the-art TTS models with extensions like full-band E2E text-to-waveform modeling.
  arXiv  Detail & Related papers  (2021-10-15T03:27:45Z)
• Improving Text Generation with Student-Forcing Optimal Transport [122.11881937642401]
  We propose using optimal transport (OT) to match the sequences generated in training and testing modes. An extension is also proposed to improve the OT learning, based on the structural and contextual information of the text sequences. The effectiveness of the proposed method is validated on machine translation, text summarization, and text generation tasks.
  arXiv  Detail & Related papers  (2020-10-12T19:42:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.