Iterative Self-Learning for Expressive Text-to-Speech Synthesis

• URL: http://arxiv.org/abs/2608.15910v1
• Date: Sun, 16 Aug 2026 19:54:20 GMT
• Title: Iterative Self-Learning for Expressive Text-to-Speech Synthesis
• Abstract summary: We propose an Iterative Self-Learning framework for expressive text-to-speech.<n>It iteratively pseudo-labels unlabeled speech using the current model, retrains on the combined labeled and pseudo-labeled data, and repeats.<n>We find that iterative refinement can improve pseudo-label accuracy over single-pass baselines.
• Score: 20.391383791266342
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Expressive text-to-speech (TTS) systems that use explicit conditioning labels provide direct and interpretable control over expressive attributes, in contrast to reference-based or prompting-based approaches, but require labeled data. Obtaining these labels at scale is costly and time-consuming, yet no prior semi-supervised framework addresses this specific bottleneck. Existing semi-supervised TTS methods instead target scarcity of paired speech-text data or transcriptions. To address the scarcity of expressive labels, we propose an Iterative Self-Learning (ISL) framework for expressive TTS, built on Invert-Classify, a classifier-free method that recovers discrete expressive labels by inverting a frozen generative model. The framework iteratively pseudo-labels unlabeled speech using the current model, retrains on the combined labeled and pseudo-labeled data, and repeats, progressively refining label quality and synthesis. We validate on two expressive tasks, word-level prominence and utterance-level emotion, across multiple low-resource data splits. We find that iterative refinement can improve pseudo-label accuracy over single-pass baselines. Furthermore, we observe that these improvements in pseudo-labeling of expressivity translate to gains in expressive label adherence and synthesis quality, confirmed by objective metrics and human listening tests. In the most data-scarce conditions, ISL-trained models outperform single-pass pseudo-labeling and further approach fully supervised performance, demonstrating that gradient-based ISL is an effective solution to expressive label scarcity in low-resource TTS.

Related papers

• Boosting Meta-Learning for Few-Shot Text Classification via Label-guided Distance Scaling [14.128153594493964]
  Few-shot text classification aims to recognize unseen classes with limited labeled text samples.<n>We propose a textbfLabel-guided textbfDistance textbfScaling (LDS) strategy.<n>In the training stage, we design a label-guided loss to inject label semantic information, pulling closer the sample representations.<n>In the testing stage, we propose a Label-guided Scaler which scales sample representations with label semantics to provide additional supervision signals.
  arXiv  Detail & Related papers  (2026-02-28T13:59:15Z)
• BiRQ: Bi-Level Self-Labeling Random Quantization for Self-Supervised Speech Recognition [63.45645200463539]
  BiRQ is a bilevel SSL framework that combines the efficiency of BEST-RQ with the refinement benefits of HuBERT-style label enhancement.<n>We validate our method on various datasets, including 960-hour LibriSpeech, 150-hour AMI meetings and 5,000-hour YODAS.
  arXiv  Detail & Related papers  (2025-09-18T21:09:29Z)
• Zero-Shot Pseudo Labels Generation Using SAM and CLIP for Semi-Supervised Semantic Segmentation [0.0]
  We propose a method to train a semantic segmentation model using images with annotated labels and pseudo labels.<n>The accuracy of the model depends on the quality of the pseudo labels and the amount of data with annotated labels.<n>The effectiveness of the proposed method is demonstrated through the experiments using the public datasets: PASCAL and MS COCO.
  arXiv  Detail & Related papers  (2025-05-26T11:31:13Z)
• Semi-Supervised Cognitive State Classification from Speech with Multi-View Pseudo-Labeling [21.82879779173242]
  The lack of labeled data is a common challenge in speech classification tasks.<n>We propose a Semi-Supervised Learning (SSL) framework, introducing a novel multi-view pseudo-labeling method.<n>We evaluate our SSL framework on emotion recognition and dementia detection tasks.
  arXiv  Detail & Related papers  (2024-09-25T13:51:19Z)
• Prompt-based Pseudo-labeling Strategy for Sample-Efficient Semi-Supervised Extractive Summarization [12.582774521907227]
  Semi-supervised learning (SSL) is a widely used technique in scenarios where labeled data is scarce and unlabeled data is abundant. Standard SSL methods follow a teacher-student paradigm to first train a classification model and then use the classifier's confidence values to select pseudo-labels. We propose a prompt-based pseudo-labeling strategy with LLMs that picks unlabeled examples with more accurate pseudo-labels.
  arXiv  Detail & Related papers  (2023-11-16T04:29:41Z)
• LanSER: Language-Model Supported Speech Emotion Recognition [25.597250907836152]
  We present LanSER, a method that enables the use of unlabeled data by inferring weak emotion labels via pre-trained large language models. For inferring weak labels constrained to a taxonomy, we use a textual entailment approach that selects an emotion label with the highest entailment score for a speech transcript extracted via automatic speech recognition. Our experimental results show that models pre-trained on large datasets with this weak supervision outperform other baseline models on standard SER datasets when fine-tuned, and show improved label efficiency.
  arXiv  Detail & Related papers  (2023-09-07T19:21:08Z)
• Seq-UPS: Sequential Uncertainty-aware Pseudo-label Selection for Semi-Supervised Text Recognition [21.583569162994277]
  One of the most popular SSL approaches is pseudo-labeling (PL) PL methods are severely degraded by noise and are prone to over-fitting to noisy labels. We propose a pseudo-label generation and an uncertainty-based data selection framework for text recognition.
  arXiv  Detail & Related papers  (2022-08-31T02:21:02Z)
• Transductive CLIP with Class-Conditional Contrastive Learning [68.51078382124331]
  We propose Transductive CLIP, a novel framework for learning a classification network with noisy labels from scratch. A class-conditional contrastive learning mechanism is proposed to mitigate the reliance on pseudo labels. ensemble labels is adopted as a pseudo label updating strategy to stabilize the training of deep neural networks with noisy labels.
  arXiv  Detail & Related papers  (2022-06-13T14:04:57Z)
• S3: Supervised Self-supervised Learning under Label Noise [53.02249460567745]
  In this paper we address the problem of classification in the presence of label noise. In the heart of our method is a sample selection mechanism that relies on the consistency between the annotated label of a sample and the distribution of the labels in its neighborhood in the feature space. Our method significantly surpasses previous methods on both CIFARCIFAR100 with artificial noise and real-world noisy datasets such as WebVision and ANIMAL-10N.
  arXiv  Detail & Related papers  (2021-11-22T15:49:20Z)
• Semi-Supervised Speech Recognition via Graph-based Temporal Classification [59.58318952000571]
  Semi-supervised learning has demonstrated promising results in automatic speech recognition by self-training. The effectiveness of this approach largely relies on the pseudo-label accuracy. Alternative ASR hypotheses of an N-best list can provide more accurate labels for an unlabeled speech utterance.
  arXiv  Detail & Related papers  (2020-10-29T14:56:56Z)
• PseudoSeg: Designing Pseudo Labels for Semantic Segmentation [78.35515004654553]
  We present a re-design of pseudo-labeling to generate structured pseudo labels for training with unlabeled or weakly-labeled data. We demonstrate the effectiveness of the proposed pseudo-labeling strategy in both low-data and high-data regimes.
  arXiv  Detail & Related papers  (2020-10-19T17:59:30Z)
• Semi-supervised Relation Extraction via Incremental Meta Self-Training [56.633441255756075]
  Semi-Supervised Relation Extraction methods aim to leverage unlabeled data in addition to learning from limited samples. Existing self-training methods suffer from the gradual drift problem, where noisy pseudo labels on unlabeled data are incorporated during training. We propose a method called MetaSRE, where a Relation Label Generation Network generates quality assessment on pseudo labels by (meta) learning from the successful and failed attempts on Relation Classification Network as an additional meta-objective.
  arXiv  Detail & Related papers  (2020-10-06T03:54:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.