CapSpeech: Enabling Downstream Applications in Style-Captioned Text-to-Speech

• URL: http://arxiv.org/abs/2506.02863v1
• Date: Tue, 03 Jun 2025 13:28:55 GMT
• Title: CapSpeech: Enabling Downstream Applications in Style-Captioned Text-to-Speech
• Authors: Helin Wang, Jiarui Hai, Dading Chong, Karan Thakkar, Tiantian Feng, Dongchao Yang, Junhyeok Lee, Laureano Moro Velazquez, Jesus Villalba, Zengyi Qin, Shrikanth Narayanan, Mounya Elhiali, Najim Dehak,
• Abstract summary: We introduce CapSpeech, a new benchmark for a series of CapTTS-related tasks.<n>CapSpeech comprises over 10 million machine-annotated audio-caption pairs and nearly 0.36 million human-annotated audio-caption pairs.<n>We conduct comprehensive experiments using both autoregressive and non-autoregressive models on CapSpeech.
• Score: 51.439665930861146
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Recent advancements in generative artificial intelligence have significantly transformed the field of style-captioned text-to-speech synthesis (CapTTS). However, adapting CapTTS to real-world applications remains challenging due to the lack of standardized, comprehensive datasets and limited research on downstream tasks built upon CapTTS. To address these gaps, we introduce CapSpeech, a new benchmark designed for a series of CapTTS-related tasks, including style-captioned text-to-speech synthesis with sound events (CapTTS-SE), accent-captioned TTS (AccCapTTS), emotion-captioned TTS (EmoCapTTS), and text-to-speech synthesis for chat agent (AgentTTS). CapSpeech comprises over 10 million machine-annotated audio-caption pairs and nearly 0.36 million human-annotated audio-caption pairs. In addition, we introduce two new datasets collected and recorded by a professional voice actor and experienced audio engineers, specifically for the AgentTTS and CapTTS-SE tasks. Alongside the datasets, we conduct comprehensive experiments using both autoregressive and non-autoregressive models on CapSpeech. Our results demonstrate high-fidelity and highly intelligible speech synthesis across a diverse range of speaking styles. To the best of our knowledge, CapSpeech is the largest available dataset offering comprehensive annotations for CapTTS-related tasks. The experiments and findings further provide valuable insights into the challenges of developing CapTTS systems.

Related papers

• UmbraTTS: Adapting Text-to-Speech to Environmental Contexts with Flow Matching [17.559310386487493]
  We introduce UmbraTTS, a flow-matching based TTS model that jointly generates both speech and environmental audio.<n>Our model allows fine-grained control over background volume and produces diverse, coherent, and context-aware audio scenes.
  arXiv  Detail & Related papers  (2025-06-11T15:43:08Z)
• Scaling Rich Style-Prompted Text-to-Speech Datasets [49.1662517033426]
  Paralinguistic Speech Captions (ParaSpeechCaps) is a large-scale dataset that annotates speech utterances with rich style captions.<n>ParaSpeechCaps covers a total of 59 style tags, including both speaker-level intrinsic tags and utterance-level situational tags.<n>It consists of 342 hours of human-labelled data and 2427 hours of automatically annotated data.
  arXiv  Detail & Related papers  (2025-03-06T18:57:40Z)
• Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokens [31.575335190916995]
  We introduce Spark-TTS, a novel system powered by BiCodec, a single-stream speech that decomposes speech into two complementary token types.<n>To facilitate research in controllable TTS, we introduce VoxBox, a meticulously curated 100,000-hour dataset with comprehensive attribute annotations.
  arXiv  Detail & Related papers  (2025-03-03T16:23:10Z)
• Visual-Aware Text-to-Speech [101.89332968344102]
  We present a new visual-aware text-to-speech (VA-TTS) task to synthesize speech conditioned on both textual inputs and visual feedback of the listener in face-to-face communication. We devise a baseline model to fuse phoneme linguistic information and listener visual signals for speech synthesis.
  arXiv  Detail & Related papers  (2023-06-21T05:11:39Z)
• WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research [82.42802570171096]
  We introduce WavCaps, the first large-scale weakly-labelled audio captioning dataset, comprising approximately 400k audio clips with paired captions. Online-harvested raw descriptions are highly noisy and unsuitable for direct use in tasks such as automated audio captioning. We propose a three-stage processing pipeline for filtering noisy data and generating high-quality captions, where ChatGPT, a large language model, is leveraged to filter and transform raw descriptions automatically.
  arXiv  Detail & Related papers  (2023-03-30T14:07:47Z)
• A Vector Quantized Approach for Text to Speech Synthesis on Real-World Spontaneous Speech [94.64927912924087]
  We train TTS systems using real-world speech from YouTube and podcasts. Recent Text-to-Speech architecture is designed for multiple code generation and monotonic alignment. We show thatRecent Text-to-Speech architecture outperforms existing TTS systems in several objective and subjective measures.
  arXiv  Detail & Related papers  (2023-02-08T17:34:32Z)
• PromptTTS: Controllable Text-to-Speech with Text Descriptions [32.647362978555485]
  We develop a text-to-speech (TTS) system that takes a prompt with both style and content descriptions as input to synthesize the corresponding speech. PromptTTS consists of a style encoder and a content encoder to extract the corresponding representations from the prompt. Experiments show that PromptTTS can generate speech with precise style control and high speech quality.
  arXiv  Detail & Related papers  (2022-11-22T10:58:38Z)
• Guided-TTS:Text-to-Speech with Untranscribed Speech [22.548875263927396]
  We present Guided-TTS, a high-quality TTS model that learns to generate speech from untranscribed speech data. For text-to-speech synthesis, we guide the generative process of the unconditional DDPM via phoneme classification to produce mel-spectrograms.
  arXiv  Detail & Related papers  (2021-11-23T10:05:05Z)
• A Survey on Neural Speech Synthesis [110.39292386792555]
  Text to speech (TTS) is a hot research topic in speech, language, and machine learning communities. We conduct a comprehensive survey on neural TTS, aiming to provide a good understanding of current research and future trends. We focus on the key components in neural TTS, including text analysis, acoustic models and vocoders, and several advanced topics, including fast TTS, low-resource TTS, robust TTS, expressive TTS, and adaptive TTS, etc.
  arXiv  Detail & Related papers  (2021-06-29T16:50:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.