StyleSinger: Style Transfer for Out-of-Domain Singing Voice Synthesis
- URL: http://arxiv.org/abs/2312.10741v2
- Date: Tue, 2 Jan 2024 12:59:20 GMT
- Title: StyleSinger: Style Transfer for Out-of-Domain Singing Voice Synthesis
- Authors: Yu Zhang, Rongjie Huang, Ruiqi Li, JinZheng He, Yan Xia, Feiyang Chen,
Xinyu Duan, Baoxing Huai, Zhou Zhao
- Abstract summary: Style transfer for out-of-domain singing voice synthesis (SVS) focuses on generating high-quality singing voices with unseen styles.
StyleSinger is the first singing voice synthesis model for zero-shot style transfer of out-of-domain reference singing voice samples.
Our evaluations in zero-shot style transfer undeniably establish that StyleSinger outperforms baseline models in both audio quality and similarity to the reference singing voice samples.
- Score: 65.56386869666025
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Style transfer for out-of-domain (OOD) singing voice synthesis (SVS) focuses
on generating high-quality singing voices with unseen styles (such as timbre,
emotion, pronunciation, and articulation skills) derived from reference singing
voice samples. However, the endeavor to model the intricate nuances of singing
voice styles is an arduous task, as singing voices possess a remarkable degree
of expressiveness. Moreover, existing SVS methods encounter a decline in the
quality of synthesized singing voices in OOD scenarios, as they rest upon the
assumption that the target vocal attributes are discernible during the training
phase. To overcome these challenges, we propose StyleSinger, the first singing
voice synthesis model for zero-shot style transfer of out-of-domain reference
singing voice samples. StyleSinger incorporates two critical approaches for
enhanced effectiveness: 1) the Residual Style Adaptor (RSA) which employs a
residual quantization module to capture diverse style characteristics in
singing voices, and 2) the Uncertainty Modeling Layer Normalization (UMLN) to
perturb the style attributes within the content representation during the
training phase and thus improve the model generalization. Our extensive
evaluations in zero-shot style transfer undeniably establish that StyleSinger
outperforms baseline models in both audio quality and similarity to the
reference singing voice samples. Access to singing voice samples can be found
at https://stylesinger.github.io/.
Related papers
- SPA-SVC: Self-supervised Pitch Augmentation for Singing Voice Conversion [12.454955437047573]
We propose a Self-supervised Pitch Augmentation method for Singing Voice Conversion (SPA-SVC)
We introduce a cycle pitch shifting training strategy and Structural Similarity Index (SSIM) loss into our SVC model, effectively enhancing its performance.
Experimental results on the public singing datasets M4Singer indicate that our proposed method significantly improves model performance.
arXiv Detail & Related papers (2024-06-09T08:34:01Z) - Prompt-Singer: Controllable Singing-Voice-Synthesis with Natural Language Prompt [50.25271407721519]
We propose Prompt-Singer, the first SVS method that enables attribute controlling on singer gender, vocal range and volume with natural language.
We adopt a model architecture based on a decoder-only transformer with a multi-scale hierarchy, and design a range-melody decoupled pitch representation.
Experiments show that our model achieves favorable controlling ability and audio quality.
arXiv Detail & Related papers (2024-03-18T13:39:05Z) - Enhancing the vocal range of single-speaker singing voice synthesis with
melody-unsupervised pre-training [82.94349771571642]
This work proposes a melody-unsupervised multi-speaker pre-training method to enhance the vocal range of the single-speaker.
It is the first to introduce a differentiable duration regulator to improve the rhythm naturalness of the synthesized voice.
Experimental results verify that the proposed SVS system outperforms the baseline on both sound quality and naturalness.
arXiv Detail & Related papers (2023-09-01T06:40:41Z) - Towards Improving the Expressiveness of Singing Voice Synthesis with
BERT Derived Semantic Information [51.02264447897833]
This paper presents an end-to-end high-quality singing voice synthesis (SVS) system that uses bidirectional encoder representation from Transformers (BERT) derived semantic embeddings.
The proposed SVS system can produce singing voice with higher-quality outperforming VISinger.
arXiv Detail & Related papers (2023-08-31T16:12:01Z) - Make-A-Voice: Unified Voice Synthesis With Discrete Representation [77.3998611565557]
Make-A-Voice is a unified framework for synthesizing and manipulating voice signals from discrete representations.
We show that Make-A-Voice exhibits superior audio quality and style similarity compared with competitive baseline models.
arXiv Detail & Related papers (2023-05-30T17:59:26Z) - Robust One-Shot Singing Voice Conversion [28.707278256253385]
High-quality singing voice conversion (SVC) of unseen singers remains challenging due to wide variety of musical expressions in pitch, loudness, and pronunciation.
We present a robust one-shot SVC that performs any-to-any SVC robustly even on distorted singing voices.
Experimental results show that the proposed method outperforms state-of-the-art one-shot SVC baselines for both seen and unseen singers.
arXiv Detail & Related papers (2022-10-20T08:47:35Z) - Unsupervised Cross-Domain Singing Voice Conversion [105.1021715879586]
We present a wav-to-wav generative model for the task of singing voice conversion from any identity.
Our method utilizes both an acoustic model, trained for the task of automatic speech recognition, together with melody extracted features to drive a waveform-based generator.
arXiv Detail & Related papers (2020-08-06T18:29:11Z) - Adversarially Trained Multi-Singer Sequence-To-Sequence Singing
Synthesizer [11.598416444452619]
We design a multi-singer framework to leverage all the existing singing data of different singers.
We incorporate an adversarial task of singer classification to make encoder output less singer dependent.
The proposed synthesizer can generate higher quality singing voice than baseline.
arXiv Detail & Related papers (2020-06-18T07:20:11Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.