Autoregressive Speech Synthesis with Next-Distribution Prediction

• URL: http://arxiv.org/abs/2412.16846v1
• Date: Sun, 22 Dec 2024 04:03:24 GMT
• Title: Autoregressive Speech Synthesis with Next-Distribution Prediction
• Authors: Xinfa Zhu, Wenjie Tian, Lei Xie,
• Abstract summary: We introduce KALL-E, a novel autoregressive (AR) language modeling approach with next-distribution prediction for text-to-speech synthesis. KALL-E directly models and predicts the continuous speech distribution conditioned on text without relying on VAE- or diffusion-based components. A single AR language model predicts these continuous speech distributions from text, with a Kullback-Leibler divergence loss as the constraint.
• Score: 7.52773630782303
• License:
• Abstract: We introduce KALL-E, a novel autoregressive (AR) language modeling approach with next-distribution prediction for text-to-speech (TTS) synthesis. Unlike existing methods, KALL-E directly models and predicts the continuous speech distribution conditioned on text without relying on VAE- or diffusion-based components. Specifically, we use WaveVAE to extract continuous speech distributions from waveforms instead of using discrete speech tokens. A single AR language model predicts these continuous speech distributions from text, with a Kullback-Leibler divergence loss as the constraint. Experimental results show that KALL-E outperforms open-source implementations of YourTTS, VALL-E, NaturalSpeech 2, and CosyVoice in terms of naturalness and speaker similarity in zero-shot TTS scenarios. Moreover, KALL-E demonstrates exceptional zero-shot capabilities in emotion and accent cloning. Importantly, KALL-E presents a more straightforward and effective paradigm for using continuous speech representations in TTS. Audio samples are available at: \url{https://zxf-icpc.github.io/kalle/}.

Related papers

• SimpleSpeech 2: Towards Simple and Efficient Text-to-Speech with Flow-based Scalar Latent Transformer Diffusion Models [64.40250409933752]
  We build upon our previous publication by implementing a simple and efficient non-autoregressive (NAR) TTS framework, termed SimpleSpeech 2. SimpleSpeech 2 effectively combines the strengths of both autoregressive (AR) and non-autoregressive (NAR) methods. We show a significant improvement in generation performance and generation speed compared to our previous work and other state-of-the-art (SOTA) large-scale TTS models.
  arXiv  Detail & Related papers  (2024-08-25T17:07:39Z)
• VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignment [101.2489492032816]
  VALL-E R is a robust and efficient zero-shot Text-to-Speech system. This research has the potential to be applied to meaningful projects, including the creation of speech for those affected by aphasia.
  arXiv  Detail & Related papers  (2024-06-12T04:09:44Z)
• RALL-E: Robust Codec Language Modeling with Chain-of-Thought Prompting for Text-to-Speech Synthesis [84.57932472551889]
  RALL-E is a robust language modeling method for text-to-speech synthesis. RALL-E improves the WER of zero-shot TTS from $5.6%$ (without reranking) to $2.5%$ and $1.0%$, respectively.
  arXiv  Detail & Related papers  (2024-04-04T05:15:07Z)
• RobustL2S: Speaker-Specific Lip-to-Speech Synthesis exploiting Self-Supervised Representations [13.995231731152462]
  We propose RobustL2S, a modularized framework for Lip-to-Speech synthesis. A non-autoregressive sequence-to-sequence model maps self-supervised visual features to a representation of disentangled speech content. A vocoder then converts the speech features into raw waveforms.
  arXiv  Detail & Related papers  (2023-07-03T09:13:57Z)
• PauseSpeech: Natural Speech Synthesis via Pre-trained Language Model and Pause-based Prosody Modeling [25.966328901566815]
  We propose PuaseSpeech, a speech synthesis system with a pre-trained language model and pause-based prosody modeling. Experimental results show PauseSpeech outperforms previous models in terms of naturalness.
  arXiv  Detail & Related papers  (2023-06-13T01:36:55Z)
• NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers [90.83782600932567]
  We develop NaturalSpeech 2, a TTS system that leverages a neural audio predictor with residual vectorizers to get the quantized latent vectors. We scale NaturalSpeech 2 to large-scale datasets with 44K hours of speech and singing data and evaluate its voice quality on unseen speakers. NaturalSpeech 2 outperforms previous TTS systems by a large margin in terms of prosody/timbre similarity, synthesis, and voice quality in a zero-shot setting.
  arXiv  Detail & Related papers  (2023-04-18T16:31:59Z)
• Unsupervised TTS Acoustic Modeling for TTS with Conditional Disentangled Sequential VAE [36.50265124324876]
  We propose a novel unsupervised text-to-speech acoustic model training scheme, named UTTS, which does not require text-audio pairs. The framework offers a flexible choice of a speaker's duration model, timbre feature (identity) and content for TTS inference. Experiments demonstrate that UTTS can synthesize speech of high naturalness and intelligibility measured by human and objective evaluations.
  arXiv  Detail & Related papers  (2022-06-06T11:51:22Z)
• ProsoSpeech: Enhancing Prosody With Quantized Vector Pre-training in Text-to-Speech [96.0009517132463]
  We introduce a word-level prosody encoder, which quantizes the low-frequency band of the speech and compresses prosody attributes in the latent prosody vector (LPV) We then introduce an LPV predictor, which predicts LPV given word sequence and fine-tune it on the high-quality TTS dataset. Experimental results show that ProsoSpeech can generate speech with richer prosody compared with baseline methods.
  arXiv  Detail & Related papers  (2022-02-16T01:42:32Z)
• Direct simultaneous speech to speech translation [29.958601064888132]
  We present the first direct simultaneous speech-to-speech translation (Simul-S2ST) model. The model can start generating translation in the target speech before consuming the full source speech content.
  arXiv  Detail & Related papers  (2021-10-15T17:59:15Z)
• Direct speech-to-speech translation with discrete units [64.19830539866072]
  We present a direct speech-to-speech translation (S2ST) model that translates speech from one language to speech in another language without relying on intermediate text generation. We propose to predict the self-supervised discrete representations learned from an unlabeled speech corpus instead. When target text transcripts are available, we design a multitask learning framework with joint speech and text training that enables the model to generate dual mode output (speech and text) simultaneously in the same inference pass.
  arXiv  Detail & Related papers  (2021-07-12T17:40:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.