Imitation Learning for Elder-Facing Speech Synthesis

• URL: http://arxiv.org/abs/2606.21053v1
• Date: Fri, 19 Jun 2026 02:36:35 GMT
• Title: Imitation Learning for Elder-Facing Speech Synthesis
• Abstract summary: We propose a novel imitation learning (IL) framework to learn text-to-speech (TTS) models from expert demonstrations.<n>We further improve Group Relative Policy Optimization with two-stage on-policy reward learning (OPRL) to mitigate reward hacking under limited supervision from expert demonstration.
• Score: 65.98429501808018
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in text-to-speech (TTS) synthesis have achieved highly natural and expressive speech generation. However, these systems are designed for general adults and overlook older adults' speech comprehension needs due to age-related sensory and cognitive decline. Prior work involves older adults by collecting preference feedback to tune model parameters. However, obtaining sufficient preference data is costly and difficult, as older adults quickly become fatigued during collection. In this paper, we propose a novel imitation learning (IL) framework to learn TTS models from expert demonstrations. We further improve Group Relative Policy Optimization (GRPO) with two-stage on-policy reward learning (OPRL) to mitigate reward hacking under limited supervision from expert demonstration. Experimental results show that GRPO w/ OPRL outperforms GRPO and supervised baselines in objective and subjective metrics. Audio samples are available at https://dongru1.github.io/demo/im-efss

Related papers

• VoiceTTA: Enhancing Zero-Shot Text-to-Speech via Reinforcement Learning-Based Test-Time Adaptation [35.0242994611344]
  VoiceTTA is a reinforcement learning-based test-time adaptation (TTA) method that improves voice imitation of pretrained zero-shot TTS models.<n>Experiments demonstrate substantial improvements on uncommon speech prompts, outperforming state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-06-25T02:18:24Z)
• Benchmarking Training Paradigms, Dataset Composition, and Model Scaling for Child ASR in ESPnet [72.53502346791814]
  We compare flat-start training across datasets, SSL representations (WavLM, XEUS), and decoder architectures.<n> SSL representations are biased toward adult speech, with flat-start training on child speech mitigating these biases.<n>Age-related ASR and speaker verification analysis highlights the limitations of proprietary models.
  arXiv  Detail & Related papers  (2025-08-22T17:59:35Z)
• OZSpeech: One-step Zero-shot Speech Synthesis with Learned-Prior-Conditioned Flow Matching [3.05024318465243]
  OZSpeech is the first TTS method to explore optimal transport conditional flow matching with one-step sampling.<n>Our approach operates on disentangled, factorized components of speech in token format, enabling accurate modeling of each speech attribute.<n> Experimental results show that our method achieves promising performance over existing methods in content accuracy, naturalness, prosody generation, and speaker style preservation.
  arXiv  Detail & Related papers  (2025-05-19T07:31:55Z)
• Reduce, Reuse, Recycle: Is Perturbed Data better than Other Language augmentation for Low Resource Self-Supervised Speech Models [48.44820587495038]
  Self-supervised representation learning (SSRL) has demonstrated superior performance than supervised models for tasks including phoneme recognition. Training SSRL models poses a challenge for low-resource languages where sufficient pre-training data may not be available. We propose to use audio augmentation techniques, namely: pitch variation, noise addition, accented target language and other language speech to pre-train SSRL models in a low resource condition and evaluate phoneme recognition.
  arXiv  Detail & Related papers  (2023-09-22T10:09:09Z)
• Leveraging Pre-trained AudioLDM for Sound Generation: A Benchmark Study [33.10311742703679]
  We make the first attempt to investigate the benefits of pre-training on sound generation with AudioLDM. Our study demonstrates the advantages of the pre-trained AudioLDM, especially in data-scarcity scenarios. We benchmark the sound generation task on various frequently-used datasets.
  arXiv  Detail & Related papers  (2023-03-07T12:49:45Z)
• Any-speaker Adaptive Text-To-Speech Synthesis with Diffusion Models [65.28001444321465]
  Grad-StyleSpeech is an any-speaker adaptive TTS framework based on a diffusion model. It can generate highly natural speech with extremely high similarity to target speakers' voice, given a few seconds of reference speech. It significantly outperforms speaker-adaptive TTS baselines on English benchmarks.
  arXiv  Detail & Related papers  (2022-11-17T07:17:24Z)
• Transfer Learning for Robust Low-Resource Children's Speech ASR with Transformers and Source-Filter Warping [11.584388304271029]
  We propose a data augmentation technique based on the source-filter model of speech to close the domain gap between adult and children's speech. Using this augmentation strategy, we apply transfer learning on a Transformer model pre-trained on adult data. This model follows the recently introduced XLS-R architecture, a wav2vec 2.0 model pre-trained on several cross-lingual adult speech corpora.
  arXiv  Detail & Related papers  (2022-06-19T12:57:47Z)
• An Exploration of Prompt Tuning on Generative Spoken Language Model for Speech Processing Tasks [112.1942546460814]
  We report the first exploration of the prompt tuning paradigm for speech processing tasks based on Generative Spoken Language Model (GSLM) Experiment results show that the prompt tuning technique achieves competitive performance in speech classification tasks with fewer trainable parameters than fine-tuning specialized downstream models.
  arXiv  Detail & Related papers  (2022-03-31T03:26:55Z)
• A Text-to-Speech Pipeline, Evaluation Methodology, and Initial Fine-Tuning Results for Child Speech Synthesis [3.2548794659022398]
  Speech synthesis has come a long way as current text-to-speech (TTS) models can now generate natural human-sounding speech. This study developed and validated a training pipeline for fine-tuning state-of-the-art neural TTS models using child speech datasets.
  arXiv  Detail & Related papers  (2022-03-22T09:34:21Z)
• An Exploration of Self-Supervised Pretrained Representations for End-to-End Speech Recognition [98.70304981174748]
  We focus on the general applications of pretrained speech representations, on advanced end-to-end automatic speech recognition (E2E-ASR) models. We select several pretrained speech representations and present the experimental results on various open-source and publicly available corpora for E2E-ASR.
  arXiv  Detail & Related papers  (2021-10-09T15:06:09Z)
• Controllable Generation from Pre-trained Language Models via Inverse Prompting [47.23315683944257]
  We propose an innovative method, inverse prompting, to better control text generation. Inverse prompting uses generated text to inversely predict the prompt during beam search. Our results show that our proposed method substantially outperforms the baselines.
  arXiv  Detail & Related papers  (2021-03-19T08:36:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.