On-device neural speech synthesis

• URL: http://arxiv.org/abs/2109.08710v1
• Date: Fri, 17 Sep 2021 18:31:31 GMT
• Title: On-device neural speech synthesis
• Authors: Sivanand Achanta, Albert Antony, Ladan Golipour, Jiangchuan Li, Tuomo Raitio, Ramya Rasipuram, Francesco Rossi, Jennifer Shi, Jaimin Upadhyay, David Winarsky, Hepeng Zhang
• Abstract summary: Tacotron and WaveRNN have made it possible to construct a fully neural network based TTS system. We present key modeling improvements and optimization strategies that enable deploying these models on GPU servers and on mobile devices. The proposed system can generate high-quality 24 kHz speech at 5x faster than real time on server and 3x faster than real time on mobile devices.
• Score: 3.716815259884143
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in text-to-speech (TTS) synthesis, such as Tacotron and WaveRNN, have made it possible to construct a fully neural network based TTS system, by coupling the two components together. Such a system is conceptually simple as it only takes grapheme or phoneme input, uses Mel-spectrogram as an intermediate feature, and directly generates speech samples. The system achieves quality equal or close to natural speech. However, the high computational cost of the system and issues with robustness have limited their usage in real-world speech synthesis applications and products. In this paper, we present key modeling improvements and optimization strategies that enable deploying these models, not only on GPU servers, but also on mobile devices. The proposed system can generate high-quality 24 kHz speech at 5x faster than real time on server and 3x faster than real time on mobile devices.

Related papers

• Lina-Speech: Gated Linear Attention is a Fast and Parameter-Efficient Learner for text-to-speech synthesis [7.2129341612013285]
  We introduce Lina-Speech, a model that replaces traditional self-attention mechanisms with emerging recurrent architectures like Gated Linear Attention (GLA) This approach is fast, easy to deploy, and achieves performance comparable to fine-tuned baselines when the dataset size ranges from 3 to 15 minutes.
  arXiv  Detail & Related papers  (2024-10-30T04:50:40Z)
• SimpleSpeech 2: Towards Simple and Efficient Text-to-Speech with Flow-based Scalar Latent Transformer Diffusion Models [64.40250409933752]
  We build upon our previous publication by implementing a simple and efficient non-autoregressive (NAR) TTS framework, termed SimpleSpeech 2. SimpleSpeech 2 effectively combines the strengths of both autoregressive (AR) and non-autoregressive (NAR) methods. We show a significant improvement in generation performance and generation speed compared to our previous work and other state-of-the-art (SOTA) large-scale TTS models.
  arXiv  Detail & Related papers  (2024-08-25T17:07:39Z)
• Pheme: Efficient and Conversational Speech Generation [52.34331755341856]
  We introduce the Pheme model series that offers compact yet high-performing conversational TTS models. It can be trained efficiently on smaller-scale conversational data, cutting data demands by more than 10x but still matching the quality of the autoregressive TTS models.
  arXiv  Detail & Related papers  (2024-01-05T14:47:20Z)
• HierSpeech++: Bridging the Gap between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-shot Speech Synthesis [39.892633589217326]
  Large language models (LLM)-based speech synthesis has been widely adopted in zero-shot speech synthesis. This paper proposes HierSpeech++, a fast and strong zero-shot speech synthesizer for text-to-speech (TTS) and voice conversion (VC)
  arXiv  Detail & Related papers  (2023-11-21T09:07:11Z)
• EfficientSpeech: An On-Device Text to Speech Model [15.118059441365343]
  State of the art (SOTA) neural text to speech (TTS) models can generate natural-sounding synthetic voices. In this work, an efficient neural TTS called EfficientSpeech that synthesizes speech on an ARM CPU in real-time is proposed.
  arXiv  Detail & Related papers  (2023-05-23T10:28:41Z)
• NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers [90.83782600932567]
  We develop NaturalSpeech 2, a TTS system that leverages a neural audio predictor with residual vectorizers to get the quantized latent vectors. We scale NaturalSpeech 2 to large-scale datasets with 44K hours of speech and singing data and evaluate its voice quality on unseen speakers. NaturalSpeech 2 outperforms previous TTS systems by a large margin in terms of prosody/timbre similarity, synthesis, and voice quality in a zero-shot setting.
  arXiv  Detail & Related papers  (2023-04-18T16:31:59Z)
• A Vector Quantized Approach for Text to Speech Synthesis on Real-World Spontaneous Speech [94.64927912924087]
  We train TTS systems using real-world speech from YouTube and podcasts. Recent Text-to-Speech architecture is designed for multiple code generation and monotonic alignment. We show thatRecent Text-to-Speech architecture outperforms existing TTS systems in several objective and subjective measures.
  arXiv  Detail & Related papers  (2023-02-08T17:34:32Z)
• fairseq S^2: A Scalable and Integrable Speech Synthesis Toolkit [60.74922995613379]
  fairseq S2 is a fairseq extension for speech synthesis. We implement a number of autoregressive (AR) and non-AR text-to-speech models, and their multi-speaker variants. To enable training speech synthesis models with less curated data, a number of preprocessing tools are built.
  arXiv  Detail & Related papers  (2021-09-14T18:20:28Z)
• Integrated Speech and Gesture Synthesis [26.267738299876314]
  Text-to-speech and co-speech gesture synthesis have until now been treated as separate areas by two different research communities. We propose to synthesize the two modalities in a single model, a new problem we call integrated speech and gesture synthesis (ISG) Model is able to achieve this with faster synthesis time and greatly reduced parameter count compared to the pipeline system.
  arXiv  Detail & Related papers  (2021-08-25T19:04:00Z)
• LightSpeech: Lightweight and Fast Text to Speech with Neural Architecture Search [127.56834100382878]
  We propose LightSpeech to automatically design more lightweight and efficient TTS models based on FastSpeech. Experiments show that the model discovered by our method achieves 15x model compression ratio and 6.5x inference speedup on CPU with on par voice quality.
  arXiv  Detail & Related papers  (2021-02-08T07:45:06Z)
• End-to-End Learning of Speech 2D Feature-Trajectory for Prosthetic Hands [0.48951183832371004]
  We propose an end-to-end convolutional neural network (CNN) that maps speech 2D features directly to trajectories for prosthetic hands. The network is written in Python with Keras library that has a corresponding backend. We optimized the CNN for NVIDIA Jetson TX2 developer kit.
  arXiv  Detail & Related papers  (2020-09-22T02:31:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.