Synthetic Speech, Real Signal: Paralinguistic Preservation and Cross-Lingual Augmentation via Voice Cloning

• URL: http://arxiv.org/abs/2607.22304v1
• Date: Fri, 24 Jul 2026 13:46:00 GMT
• Title: Synthetic Speech, Real Signal: Paralinguistic Preservation and Cross-Lingual Augmentation via Voice Cloning
• Abstract summary: We benchmark eight voice cloning models on five paralinguistic tasks across public and clinical datasets.<n>We then clone English clinical speech into Japanese and find that training on cloned data outperforms cross-lingual transfer for depression and anxiety detection.
• Score: 34.17169366465841
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Synthetic data augmentation in speech is common practice for linguistic tasks like ASR, but has seen far less work for paralinguistic ones, especially clinical tasks where labelled data is expensive and some patient groups are underrepresented. Voice cloning is one such augmentation approach, but is typically evaluated on speech intelligibility (WER) or speaker similarity (SS) rather than on downstream performance, and it remains unclear whether these preserve the paralinguistic signal such tasks depend on. We benchmark eight voice cloning models on five paralinguistic tasks across public and clinical datasets, showing most preserve signal with modest degradation. We then clone English clinical speech into Japanese and find that training on cloned data outperforms raw cross-lingual transfer for depression and anxiety detection on real Japanese speech, suggesting voice cloning is a promising direction for augmenting clinical speech data in low-resource languages.

Related papers

• Can Large Language Models Imitate Human Speech for Clinical Assessment? LLM-Driven Data Augmentation for Cognitive Score Prediction [6.724673957306738]
  We propose a large language model (LLM)-driven data augmentation framework to improve the prediction of cognitive scores from speech.<n> Experiments are conducted on a Japanese corpus in which each participant provides both a spontaneous oral narrative and a written response to the same clinical prompt.
  arXiv  Detail & Related papers  (2026-05-15T15:39:22Z)
• Adapting Self-Supervised Speech Representations for Cross-lingual Dysarthria Detection in Parkinson's Disease [72.0406069194794]
  Speech representations often encode language-dependent structure that can confound dysarthria detection.<n>We propose a representation-level language shift that aligns source-language self-supervised speech representations with the target-language distribution.<n>We evaluate the approach on oral DDK recordings from Parkinson's disease speech datasets in Czech, German, and Spanish.
  arXiv  Detail & Related papers  (2026-03-23T17:23:39Z)
• SITA: Learning Speaker-Invariant and Tone-Aware Speech Representations for Low-Resource Tonal Languages [11.655315357810371]
  SITA is a lightweight adaptation recipe that enforces Speaker-Invariance and Tone-Awareness for pretrained wav2vec-style encoders.<n>We evaluate primarily on Hmong, a highly tonal and severely under-the-shelf multilingual encoders fail to represent tone effectively.
  arXiv  Detail & Related papers  (2026-01-14T00:42:27Z)
• Towards Inclusive ASR: Investigating Voice Conversion for Dysarthric Speech Recognition in Low-Resource Languages [49.31519786009296]
  We fine-tune a voice conversion model on English dysarthric speech (UASpeech) to encode both speaker characteristics and prosodic distortions.<n>We then apply it to convert healthy non-English speech (FLEURS) into non-English dysarthric-like speech.<n>The generated data is then used to fine-tune a multilingual ASR model, Massively Multilingually Speech (MMS)
  arXiv  Detail & Related papers  (2025-05-20T20:03:45Z)
• Voice Cloning for Dysarthric Speech Synthesis: Addressing Data Scarcity in Speech-Language Pathology [0.0]
  This study explores voice cloning to generate synthetic speech replicating the unique patterns of individuals with dysarthria.<n>Using the TORGO dataset, we address data scarcity and privacy challenges in speech-language pathology.<n>We cloned voices from dysarthric and control speakers using a commercial platform, ensuring gender-matched synthetic voices.
  arXiv  Detail & Related papers  (2025-03-03T07:44:49Z)
• Cross-Lingual Transfer Learning for Speech Translation [7.802021866251242]
  This paper examines how to expand the speech translation capability of speech foundation models with restricted data.<n>Whisper, a speech foundation model with strong performance on speech recognition and English translation, is used as the example model.<n>Using speech-to-speech retrieval to analyse the audio representations generated by the encoder, we show that utterances from different languages are mapped to a shared semantic space.
  arXiv  Detail & Related papers  (2024-07-01T09:51:48Z)
• Speech-to-Speech Translation with Discrete-Unit-Based Style Transfer [53.72998363956454]
  Direct speech-to-speech translation (S2ST) with discrete self-supervised representations has achieved remarkable accuracy. The scarcity of high-quality speaker-parallel data poses a challenge for learning style transfer during translation. We design an S2ST pipeline with style-transfer capability on the basis of discrete self-supervised speech representations and timbre units.
  arXiv  Detail & Related papers  (2023-09-14T09:52:08Z)
• Learning Cross-lingual Visual Speech Representations [108.68531445641769]
  Cross-lingual self-supervised visual representation learning has been a growing research topic in the last few years. We use the recently-proposed Raw Audio-Visual Speechs (RAVEn) framework to pre-train an audio-visual model with unlabelled data. Our experiments show that: (1) multi-lingual models with more data outperform monolingual ones, but, when keeping the amount of data fixed, monolingual models tend to reach better performance.
  arXiv  Detail & Related papers  (2023-03-14T17:05:08Z)
• Zero-Shot Cross-lingual Aphasia Detection using Automatic Speech Recognition [3.2631198264090746]
  Aphasia is a common speech and language disorder, typically caused by a brain injury or a stroke, that affects millions of people worldwide. We propose an end-to-end pipeline using pre-trained Automatic Speech Recognition (ASR) models that share cross-lingual speech representations.
  arXiv  Detail & Related papers  (2022-04-01T14:05:02Z)
• Meta-Voice: Fast few-shot style transfer for expressive voice cloning using meta learning [37.73490851004852]
  Task of few-shot style transfer for voice cloning in text-to-speech (TTS) synthesis aims at transferring speaking styles of an arbitrary source speaker to a target speaker's voice using very limited amount of neutral data. This is a very challenging task since the learning algorithm needs to deal with few-shot voice cloning and speaker-prosody disentanglement at the same time. In this paper, we approach to the hard fast few-shot style transfer for voice cloning task using meta learning.
  arXiv  Detail & Related papers  (2021-11-14T01:30:37Z)
• Cross-lingual Transfer for Speech Processing using Acoustic Language Similarity [81.51206991542242]
  Cross-lingual transfer offers a compelling way to help bridge this digital divide. Current cross-lingual algorithms have shown success in text-based tasks and speech-related tasks over some low-resource languages. We propose a language similarity approach that can efficiently identify acoustic cross-lingual transfer pairs across hundreds of languages.
  arXiv  Detail & Related papers  (2021-11-02T01:55:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.