LoRP-TTS: Low-Rank Personalized Text-To-Speech

• URL: http://arxiv.org/abs/2502.07562v1
• Date: Tue, 11 Feb 2025 14:00:12 GMT
• Title: LoRP-TTS: Low-Rank Personalized Text-To-Speech
• Authors: Łukasz Bondaruk, Jakub Kubiak,
• Abstract summary: Speech synthesis models convert written text into natural-sounding audio. Low-Rank Adaptation (LoRA) allows us to successfully use even single recordings of spontaneous speech in noisy environments as prompts.
• Score: 0.0
• License:
• Abstract: Speech synthesis models convert written text into natural-sounding audio. While earlier models were limited to a single speaker, recent advancements have led to the development of zero-shot systems that generate realistic speech from a wide range of speakers using their voices as additional prompts. However, they still struggle with imitating non-studio-quality samples that differ significantly from the training datasets. In this work, we demonstrate that utilizing Low-Rank Adaptation (LoRA) allows us to successfully use even single recordings of spontaneous speech in noisy environments as prompts. This approach enhances speaker similarity by up to $30pp$ while preserving content and naturalness. It represents a significant step toward creating truly diverse speech corpora, that is crucial in all speech-related tasks.

Related papers

• IntrinsicVoice: Empowering LLMs with Intrinsic Real-time Voice Interaction Abilities [55.11130688075417]
  We introduce IntrinsicVoic,e an LLM designed with intrinsic real-time voice interaction capabilities. Our novelty architecture, GroupFormer, can reduce speech sequences to lengths comparable to text sequences. We construct a multi-turn speech-to-speech dialogue dataset named method-500k which includes nearly 500k turns of speech-to-speech dialogues.
  arXiv  Detail & Related papers  (2024-10-09T05:04:31Z)
• Multi-modal Adversarial Training for Zero-Shot Voice Cloning [9.823246184635103]
  We propose a Transformer encoder-decoder architecture to conditionally discriminate between real and generated speech features. We introduce our novel adversarial training technique by applying it to a FastSpeech2 acoustic model and training on Libriheavy, a large multi-speaker dataset. Our model achieves improvements over the baseline in terms of speech quality and speaker similarity.
  arXiv  Detail & Related papers  (2024-08-28T16:30:41Z)
• SpeechPrompt: Prompting Speech Language Models for Speech Processing Tasks [94.10497337235083]
  We are first to explore the potential of prompting speech LMs in the domain of speech processing. We reformulate speech processing tasks into speech-to-unit generation tasks. We show that the prompting method can achieve competitive performance compared to the strong fine-tuning method.
  arXiv  Detail & Related papers  (2024-08-23T13:00:10Z)
• PromptTTS++: Controlling Speaker Identity in Prompt-Based Text-to-Speech Using Natural Language Descriptions [21.15647416266187]
  We propose PromptTTS++, a prompt-based text-to-speech (TTS) synthesis system that allows control over speaker identity using natural language descriptions. We introduce the concept of speaker prompt, which describes voice characteristics designed to be approximately independent of speaking style. Our subjective evaluation results show that the proposed method can better control speaker characteristics than the methods without the speaker prompt.
  arXiv  Detail & Related papers  (2023-09-15T04:11:37Z)
• SpeechX: Neural Codec Language Model as a Versatile Speech Transformer [57.82364057872905]
  SpeechX is a versatile speech generation model capable of zero-shot TTS and various speech transformation tasks. Experimental results show SpeechX's efficacy in various tasks, including zero-shot TTS, noise suppression, target speaker extraction, speech removal, and speech editing with or without background noise.
  arXiv  Detail & Related papers  (2023-08-14T01:01:19Z)
• NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers [90.83782600932567]
  We develop NaturalSpeech 2, a TTS system that leverages a neural audio predictor with residual vectorizers to get the quantized latent vectors. We scale NaturalSpeech 2 to large-scale datasets with 44K hours of speech and singing data and evaluate its voice quality on unseen speakers. NaturalSpeech 2 outperforms previous TTS systems by a large margin in terms of prosody/timbre similarity, synthesis, and voice quality in a zero-shot setting.
  arXiv  Detail & Related papers  (2023-04-18T16:31:59Z)
• Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesis [67.73554826428762]
  We propose a novel audio-visual speech enhancement framework for high-fidelity telecommunications in AR/VR. Our approach leverages audio-visual speech cues to generate the codes of a neural speech, enabling efficient synthesis of clean, realistic speech from noisy signals.
  arXiv  Detail & Related papers  (2022-03-31T17:57:10Z)
• A$^3$T: Alignment-Aware Acoustic and Text Pretraining for Speech Synthesis and Editing [31.666920933058144]
  We propose our framework, Alignment-Aware Acoustic-Text Pretraining (A$3$T), which reconstructs masked acoustic signals with text input and acoustic-text alignment during training. Experiments show A$3$T outperforms SOTA models on speech editing, and improves multi-speaker speech synthesis without the external speaker verification model.
  arXiv  Detail & Related papers  (2022-03-18T01:36:25Z)
• Meta-StyleSpeech : Multi-Speaker Adaptive Text-to-Speech Generation [63.561944239071615]
  StyleSpeech is a new TTS model which synthesizes high-quality speech and adapts to new speakers. With SALN, our model effectively synthesizes speech in the style of the target speaker even from single speech audio. We extend it to Meta-StyleSpeech by introducing two discriminators trained with style prototypes, and performing episodic training.
  arXiv  Detail & Related papers  (2021-06-06T15:34:11Z)
• Few Shot Adaptive Normalization Driven Multi-Speaker Speech Synthesis [18.812696623555855]
  We present a novel few shot multi-speaker speech synthesis approach (FSM-SS) Given an input text and a reference speech sample of an unseen person, FSM-SS can generate speech in that person's style in a few shot manner. We demonstrate how the affine parameters of normalization help in capturing the prosodic features such as energy and fundamental frequency.
  arXiv  Detail & Related papers  (2020-12-14T04:37:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.