Scaling Rich Style-Prompted Text-to-Speech Datasets

• URL: http://arxiv.org/abs/2503.04713v1
• Date: Thu, 06 Mar 2025 18:57:40 GMT
• Title: Scaling Rich Style-Prompted Text-to-Speech Datasets
• Authors: Anuj Diwan, Zhisheng Zheng, David Harwath, Eunsol Choi,
• Abstract summary: Paralinguistic Speech Captions (ParaSpeechCaps) is a large-scale dataset that annotates speech utterances with rich style captions.<n>ParaSpeechCaps covers a total of 59 style tags, including both speaker-level intrinsic tags and utterance-level situational tags.<n>It consists of 342 hours of human-labelled data and 2427 hours of automatically annotated data.
• Score: 49.1662517033426
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce Paralinguistic Speech Captions (ParaSpeechCaps), a large-scale dataset that annotates speech utterances with rich style captions. While rich abstract tags (e.g. guttural, nasal, pained) have been explored in small-scale human-annotated datasets, existing large-scale datasets only cover basic tags (e.g. low-pitched, slow, loud). We combine off-the-shelf text and speech embedders, classifiers and an audio language model to automatically scale rich tag annotations for the first time. ParaSpeechCaps covers a total of 59 style tags, including both speaker-level intrinsic tags and utterance-level situational tags. It consists of 342 hours of human-labelled data (PSC-Base) and 2427 hours of automatically annotated data (PSC-Scaled). We finetune Parler-TTS, an open-source style-prompted TTS model, on ParaSpeechCaps, and achieve improved style consistency (+7.9% Consistency MOS) and speech quality (+15.5% Naturalness MOS) over the best performing baseline that combines existing rich style tag datasets. We ablate several of our dataset design choices to lay the foundation for future work in this space. Our dataset, models and code are released at https://github.com/ajd12342/paraspeechcaps .

Related papers

• NonverbalTTS: A Public English Corpus of Text-Aligned Nonverbal Vocalizations with Emotion Annotations for Text-to-Speech [0.0]
  NonverbalTTS (NVTTS) is a 17-hour open-access dataset annotated with 10 types of NVs (e.g., laughter, coughs) and 8 emotional categories.<n>We propose a comprehensive pipeline that integrates automatic speech recognition (ASR), NV tagging, emotion classification, and a fusion algorithm to merge transcriptions from multiple annotators.
  arXiv  Detail & Related papers  (2025-07-17T14:17:40Z)
• CapSpeech: Enabling Downstream Applications in Style-Captioned Text-to-Speech [51.439665930861146]
  We introduce CapSpeech, a new benchmark for a series of CapTTS-related tasks.<n>CapSpeech comprises over 10 million machine-annotated audio-caption pairs and nearly 0.36 million human-annotated audio-caption pairs.<n>We conduct comprehensive experiments using both autoregressive and non-autoregressive models on CapSpeech.
  arXiv  Detail & Related papers  (2025-06-03T13:28:55Z)
• Detecting Document-level Paraphrased Machine Generated Content: Mimicking Human Writing Style and Involving Discourse Features [57.34477506004105]
  Machine-generated content poses challenges such as academic plagiarism and the spread of misinformation.<n>We introduce novel methodologies and datasets to overcome these challenges.<n>We propose MhBART, an encoder-decoder model designed to emulate human writing style.<n>We also propose DTransformer, a model that integrates discourse analysis through PDTB preprocessing to encode structural features.
  arXiv  Detail & Related papers  (2024-12-17T08:47:41Z)
• Synth4Kws: Synthesized Speech for User Defined Keyword Spotting in Low Resource Environments [8.103855990028842]
  We introduce Synth4Kws - a framework to leverage Text to Speech (TTS) synthesized data for custom KWS. We found increasing TTS phrase diversity and utterance sampling monotonically improves model performance. Our experiments are based on English and single word utterances but the findings generalize to i18n languages.
  arXiv  Detail & Related papers  (2024-07-23T21:05:44Z)
• Natural language guidance of high-fidelity text-to-speech with synthetic annotations [13.642358232817342]
  We propose a scalable method for labeling various aspects of speaker identity, style, and recording conditions. We then apply this method to a 45k hour dataset, which we use to train a speech language model. Our results demonstrate high-fidelity speech generation in a diverse range of accents, prosodic styles, channel conditions, and acoustic conditions.
  arXiv  Detail & Related papers  (2024-02-02T21:29:34Z)
• StyleCap: Automatic Speaking-Style Captioning from Speech Based on Speech and Language Self-supervised Learning Models [17.945821635380614]
  StyleCap is a method to generate natural language descriptions of speaking styles appearing in speech. StyleCap is trained with paired data of speech and natural language descriptions.
  arXiv  Detail & Related papers  (2023-11-28T04:49:17Z)
• TextrolSpeech: A Text Style Control Speech Corpus With Codec Language Text-to-Speech Models [51.529485094900934]
  We propose TextrolSpeech, which is the first large-scale speech emotion dataset annotated with rich text attributes. We introduce a multi-stage prompt programming approach that effectively utilizes the GPT model for generating natural style descriptions in large volumes. To address the need for generating audio with greater style diversity, we propose an efficient architecture called Salle.
  arXiv  Detail & Related papers  (2023-08-28T09:06:32Z)
• WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research [82.42802570171096]
  We introduce WavCaps, the first large-scale weakly-labelled audio captioning dataset, comprising approximately 400k audio clips with paired captions. Online-harvested raw descriptions are highly noisy and unsuitable for direct use in tasks such as automated audio captioning. We propose a three-stage processing pipeline for filtering noisy data and generating high-quality captions, where ChatGPT, a large language model, is leveraged to filter and transform raw descriptions automatically.
  arXiv  Detail & Related papers  (2023-03-30T14:07:47Z)
• Self-supervised Context-aware Style Representation for Expressive Speech Synthesis [23.460258571431414]
  We propose a novel framework for learning style representation from plain text in a self-supervised manner. It leverages an emotion lexicon and uses contrastive learning and deep clustering. Our method achieves improved results according to subjective evaluations on both in-domain and out-of-domain test sets in audiobook speech.
  arXiv  Detail & Related papers  (2022-06-25T05:29:48Z)
• Generating More Pertinent Captions by Leveraging Semantics and Style on Multi-Source Datasets [56.018551958004814]
  This paper addresses the task of generating fluent descriptions by training on a non-uniform combination of data sources. Large-scale datasets with noisy image-text pairs provide a sub-optimal source of supervision. We propose to leverage and separate semantics and descriptive style through the incorporation of a style token and keywords extracted through a retrieval component.
  arXiv  Detail & Related papers  (2021-11-24T19:00:05Z)
• Sentiment analysis in tweets: an assessment study from classical to modern text representation models [59.107260266206445]
  Short texts published on Twitter have earned significant attention as a rich source of information. Their inherent characteristics, such as the informal, and noisy linguistic style, remain challenging to many natural language processing (NLP) tasks. This study fulfils an assessment of existing language models in distinguishing the sentiment expressed in tweets by using a rich collection of 22 datasets.
  arXiv  Detail & Related papers  (2021-05-29T21:05:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.