TCSinger 2: Customizable Multilingual Zero-shot Singing Voice Synthesis

• URL: http://arxiv.org/abs/2505.14910v3
• Date: Fri, 30 May 2025 10:20:59 GMT
• Title: TCSinger 2: Customizable Multilingual Zero-shot Singing Voice Synthesis
• Authors: Yu Zhang, Wenxiang Guo, Changhao Pan, Dongyu Yao, Zhiyuan Zhu, Ziyue Jiang, Yuhan Wang, Tao Jin, Zhou Zhao,
• Abstract summary: multilingual zero-shot singing voice (SVS) has various potential applications in music composition and short video dubbing.<n>We introduce TCSinger 2, a multi-task multilingual zero-shot SVS model with style transfer and style control based on various prompts.<n>Results show that TCSinger 2 outperforms baseline models in both subjective and objective metrics across multiple related tasks.
• Score: 45.25653283957727
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Customizable multilingual zero-shot singing voice synthesis (SVS) has various potential applications in music composition and short video dubbing. However, existing SVS models overly depend on phoneme and note boundary annotations, limiting their robustness in zero-shot scenarios and producing poor transitions between phonemes and notes. Moreover, they also lack effective multi-level style control via diverse prompts. To overcome these challenges, we introduce TCSinger 2, a multi-task multilingual zero-shot SVS model with style transfer and style control based on various prompts. TCSinger 2 mainly includes three key modules: 1) Blurred Boundary Content (BBC) Encoder, predicts duration, extends content embedding, and applies masking to the boundaries to enable smooth transitions. 2) Custom Audio Encoder, uses contrastive learning to extract aligned representations from singing, speech, and textual prompts. 3) Flow-based Custom Transformer, leverages Cus-MOE, with F0 supervision, enhancing both the synthesis quality and style modeling of the generated singing voice. Experimental results show that TCSinger 2 outperforms baseline models in both subjective and objective metrics across multiple related tasks. Singing voice samples are available at https://aaronz345.github.io/TCSinger2Demo/.

Related papers

• Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokens [31.575335190916995]
  We introduce Spark-TTS, a novel system powered by BiCodec, a single-stream speech that decomposes speech into two complementary token types.<n>To facilitate research in controllable TTS, we introduce VoxBox, a meticulously curated 100,000-hour dataset with comprehensive attribute annotations.
  arXiv  Detail & Related papers  (2025-03-03T16:23:10Z)
• TCSinger: Zero-Shot Singing Voice Synthesis with Style Transfer and Multi-Level Style Control [58.96445085236971]
  Zero-shot singing voice synthesis (SVS) with style transfer and style control aims to generate high-quality singing voices with unseen timbres and styles.<n>We introduce TCSinger, the first zero-shot SVS model for style transfer across cross-lingual speech and singing styles.
  arXiv  Detail & Related papers  (2024-09-24T11:18:09Z)
• Prompt-Singer: Controllable Singing-Voice-Synthesis with Natural Language Prompt [50.25271407721519]
  We propose Prompt-Singer, the first SVS method that enables attribute controlling on singer gender, vocal range and volume with natural language.<n>We adopt a model architecture based on a decoder-only transformer with a multi-scale hierarchy, and design a range-melody decoupled pitch representation.<n>Experiments show that our model achieves favorable controlling ability and audio quality.
  arXiv  Detail & Related papers  (2024-03-18T13:39:05Z)
• StyleSinger: Style Transfer for Out-of-Domain Singing Voice Synthesis [63.18764165357298]
  Style transfer for out-of-domain singing voice synthesis (SVS) focuses on generating high-quality singing voices with unseen styles.<n>StyleSinger is the first singing voice synthesis model for zero-shot style transfer of out-of-domain reference singing voice samples.<n>Our evaluations in zero-shot style transfer undeniably establish that StyleSinger outperforms baseline models in both audio quality and similarity to the reference singing voice samples.
  arXiv  Detail & Related papers  (2023-12-17T15:26:16Z)
• BiSinger: Bilingual Singing Voice Synthesis [9.600465391545477]
  This paper presents BiSinger, a bilingual pop SVS system for English and Chinese Mandarin. We design a shared representation between Chinese and English singing voices, achieved by using the CMU dictionary with mapping rules. Experiments affirm that our language-independent representation and incorporation of related datasets enable a single model with enhanced performance in English and code-switch SVS.
  arXiv  Detail & Related papers  (2023-09-25T12:31:05Z)
• Make-A-Voice: Unified Voice Synthesis With Discrete Representation [77.3998611565557]
  Make-A-Voice is a unified framework for synthesizing and manipulating voice signals from discrete representations. We show that Make-A-Voice exhibits superior audio quality and style similarity compared with competitive baseline models.
  arXiv  Detail & Related papers  (2023-05-30T17:59:26Z)
• DeepSinger: Singing Voice Synthesis with Data Mined From the Web [194.10598657846145]
  DeepSinger is a multi-lingual singing voice synthesis system built from scratch using singing training data mined from music websites. We evaluate DeepSinger on our mined singing dataset that consists of about 92 hours data from 89 singers on three languages.
  arXiv  Detail & Related papers  (2020-07-09T07:00:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.