An Empirical Study on End-to-End Singing Voice Synthesis with Encoder-Decoder Architectures

• URL: http://arxiv.org/abs/2108.03008v1
• Date: Fri, 6 Aug 2021 08:51:16 GMT
• Title: An Empirical Study on End-to-End Singing Voice Synthesis with Encoder-Decoder Architectures
• Authors: Dengfeng Ke and Yuxing Lu and Xudong Liu and Yanyan Xu and Jing Sun and Cheng-Hao Cai
• Abstract summary: We use encoder-decoder neural models and a number of vocoders to achieve singing voice synthesis. We conduct experiments to demonstrate that the models can be trained using voice data with pitch information, lyrics and beat information.
• Score: 11.440111473570196
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the rapid development of neural network architectures and speech processing models, singing voice synthesis with neural networks is becoming the cutting-edge technique of digital music production. In this work, in order to explore how to improve the quality and efficiency of singing voice synthesis, in this work, we use encoder-decoder neural models and a number of vocoders to achieve singing voice synthesis. We conduct experiments to demonstrate that the models can be trained using voice data with pitch information, lyrics and beat information, and the trained models can produce smooth, clear and natural singing voice that is close to real human voice. As the models work in the end-to-end manner, they allow users who are not domain experts to directly produce singing voice by arranging pitches, lyrics and beats.

Related papers

• Prompt-Singer: Controllable Singing-Voice-Synthesis with Natural Language Prompt [50.25271407721519]
  We propose Prompt-Singer, the first SVS method that enables attribute controlling on singer gender, vocal range and volume with natural language. We adopt a model architecture based on a decoder-only transformer with a multi-scale hierarchy, and design a range-melody decoupled pitch representation. Experiments show that our model achieves favorable controlling ability and audio quality.
  arXiv  Detail & Related papers  (2024-03-18T13:39:05Z)
• Enhancing the vocal range of single-speaker singing voice synthesis with melody-unsupervised pre-training [82.94349771571642]
  This work proposes a melody-unsupervised multi-speaker pre-training method to enhance the vocal range of the single-speaker. It is the first to introduce a differentiable duration regulator to improve the rhythm naturalness of the synthesized voice. Experimental results verify that the proposed SVS system outperforms the baseline on both sound quality and naturalness.
  arXiv  Detail & Related papers  (2023-09-01T06:40:41Z)
• Human Voice Pitch Estimation: A Convolutional Network with Auto-Labeled and Synthetic Data [0.0]
  We present a specialized convolutional neural network designed for pitch extraction. Our approach combines synthetic data with auto-labeled acapella sung audio, creating a robust training environment. This work paves the way for enhanced pitch extraction in both music and voice settings.
  arXiv  Detail & Related papers  (2023-08-14T14:26:52Z)
• HiddenSinger: High-Quality Singing Voice Synthesis via Neural Audio Codec and Latent Diffusion Models [25.966328901566815]
  We propose HiddenSinger, a high-quality singing voice synthesis system using neural audio and latent diffusion models. In addition, our proposed model is extended to an unsupervised singing voice learning framework, HiddenSinger-U, to train the model. Experimental results demonstrate that our model outperforms previous models in terms of audio quality.
  arXiv  Detail & Related papers  (2023-06-12T01:21:41Z)
• A Melody-Unsupervision Model for Singing Voice Synthesis [9.137554315375919]
  We propose a melody-unsupervision model that requires only audio-and-lyrics pairs without temporal alignment in training time. We show that the proposed model is capable of being trained with speech audio and text labels but can generate singing voice in inference time.
  arXiv  Detail & Related papers  (2021-10-13T07:42:35Z)
• Sinsy: A Deep Neural Network-Based Singing Voice Synthesis System [25.573552964889963]
  This paper presents Sinsy, a deep neural network (DNN)-based singing voice synthesis (SVS) system. The proposed system is composed of four modules: a time-lag model, a duration model, an acoustic model, and a vocoder. Experimental results show our system can synthesize a singing voice with better timing, more natural vibrato, and correct pitch.
  arXiv  Detail & Related papers  (2021-08-05T17:59:58Z)
• VAW-GAN for Singing Voice Conversion with Non-parallel Training Data [81.79070894458322]
  We propose a singing voice conversion framework based on VAW-GAN. We train an encoder to disentangle singer identity and singing prosody (F0) from phonetic content. By conditioning on singer identity and F0, the decoder generates output spectral features with unseen target singer identity.
  arXiv  Detail & Related papers  (2020-08-10T09:44:10Z)
• Unsupervised Cross-Domain Singing Voice Conversion [105.1021715879586]
  We present a wav-to-wav generative model for the task of singing voice conversion from any identity. Our method utilizes both an acoustic model, trained for the task of automatic speech recognition, together with melody extracted features to drive a waveform-based generator.
  arXiv  Detail & Related papers  (2020-08-06T18:29:11Z)
• DeepSinger: Singing Voice Synthesis with Data Mined From the Web [194.10598657846145]
  DeepSinger is a multi-lingual singing voice synthesis system built from scratch using singing training data mined from music websites. We evaluate DeepSinger on our mined singing dataset that consists of about 92 hours data from 89 singers on three languages.
  arXiv  Detail & Related papers  (2020-07-09T07:00:48Z)
• Score and Lyrics-Free Singing Voice Generation [48.55126268721948]
  We explore a novel yet challenging alternative: singing voice generation without pre-assigned scores and lyrics, in both training and inference time. We implement such models using generative adversarial networks and evaluate them both objectively and subjectively.
  arXiv  Detail & Related papers  (2019-12-26T01:45:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.