Vevo2: Bridging Controllable Speech and Singing Voice Generation via Unified Prosody Learning

• URL: http://arxiv.org/abs/2508.16332v1
• Date: Fri, 22 Aug 2025 12:20:11 GMT
• Title: Vevo2: Bridging Controllable Speech and Singing Voice Generation via Unified Prosody Learning
• Authors: Xueyao Zhang, Junan Zhang, Yuancheng Wang, Chaoren Wang, Yuanzhe Chen, Dongya Jia, Zhuo Chen, Zhizheng Wu,
• Abstract summary: This paper introduces Vevo2, a unified framework for controllable speech and singing voice generation.<n>Vevo2 consists of an auto-regressive (AR) content-style modeling stage, which aims to enable controllability over text, prosody, and style.<n> Experimental results show that the unified modeling in Vevo2 brings mutual benefits to both speech and singing voice generation.
• Score: 19.2319862239494
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Controllable human voice generation, particularly for expressive domains like singing, remains a significant challenge. This paper introduces Vevo2, a unified framework for controllable speech and singing voice generation. To tackle issues like the scarcity of annotated singing data and to enable flexible controllability, Vevo2 introduces two audio tokenizers: (1) a music-notation-free prosody tokenizer that captures prosody and melody from speech, singing, and even instrumental sounds, and (2) a low-frame-rate (12.5 Hz) content-style tokenizer that encodes linguistic content, prosody, and style for both speech and singing, while enabling timbre disentanglement. Vevo2 consists of an auto-regressive (AR) content-style modeling stage, which aims to enable controllability over text, prosody, and style, as well as a flow-matching acoustic modeling stage that allows for timbre control. Particularly, during pre-training of the AR model, we propose both explicit and implicit prosody learning strategies to bridge speech and singing voice. Moreover, to further enhance the AR model's ability to follow text and prosody, we design a multi-objective post-training task that integrates both intelligibility and prosody similarity alignment. Experimental results show that the unified modeling in Vevo2 brings mutual benefits to both speech and singing voice generation. Additionally, Vevo2's effectiveness across a wide range of synthesis, conversion, and editing tasks for both speech and singing further demonstrates its strong generalization ability and versatility. Audio samples are are available at https://versasinger.github.io/.

Related papers

• Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement [18.669597283434662]
  Vevo is a versatile zero-shot voice imitation framework with controllable timbre and style.<n>To obtain the content and content-style tokens of speech, we design a fully self-supervised approach.<n>Vevo matches or surpasses existing methods in accent and emotion conversion tasks.
  arXiv  Detail & Related papers  (2025-02-11T04:18:33Z)
• S2Cap: A Benchmark and a Baseline for Singing Style Captioning [12.515874333424929]
  We present S2Cap, a dataset of singing voices with detailed descriptions covering diverse vocal, acoustic, and demographic characteristics.<n>Using this dataset, we develop an efficient and straightforward baseline algorithm for singing style captioning.
  arXiv  Detail & Related papers  (2024-09-15T21:19:24Z)
• vec2wav 2.0: Advancing Voice Conversion via Discrete Token Vocoders [26.00129172101188]
  We propose a new speech discrete token vocoder, vec2wav 2.0, which advances voice conversion (VC)<n>We use discrete tokens from speech self-supervised models as the content features of source speech, and treat VC as a prompted vocoding task.<n>We show vec2wav 2.0 achieves competitive cross-lingual VC even only trained on monolingual corpus.
  arXiv  Detail & Related papers  (2024-09-03T15:41:07Z)
• Prompt-Singer: Controllable Singing-Voice-Synthesis with Natural Language Prompt [50.25271407721519]
  We propose Prompt-Singer, the first SVS method that enables attribute controlling on singer gender, vocal range and volume with natural language.<n>We adopt a model architecture based on a decoder-only transformer with a multi-scale hierarchy, and design a range-melody decoupled pitch representation.<n>Experiments show that our model achieves favorable controlling ability and audio quality.
  arXiv  Detail & Related papers  (2024-03-18T13:39:05Z)
• StyleSinger: Style Transfer for Out-of-Domain Singing Voice Synthesis [63.18764165357298]
  Style transfer for out-of-domain singing voice synthesis (SVS) focuses on generating high-quality singing voices with unseen styles.<n>StyleSinger is the first singing voice synthesis model for zero-shot style transfer of out-of-domain reference singing voice samples.<n>Our evaluations in zero-shot style transfer undeniably establish that StyleSinger outperforms baseline models in both audio quality and similarity to the reference singing voice samples.
  arXiv  Detail & Related papers  (2023-12-17T15:26:16Z)
• Align, Adapt and Inject: Sound-guided Unified Image Generation [50.34667929051005]
  We propose a unified framework 'Align, Adapt, and Inject' (AAI) for sound-guided image generation, editing, and stylization. Our method adapts input sound into a sound token, like an ordinary word, which can plug and play with existing Text-to-Image (T2I) models. Our proposed AAI outperforms other text and sound-guided state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-06-20T12:50:49Z)
• Make-A-Voice: Unified Voice Synthesis With Discrete Representation [77.3998611565557]
  Make-A-Voice is a unified framework for synthesizing and manipulating voice signals from discrete representations. We show that Make-A-Voice exhibits superior audio quality and style similarity compared with competitive baseline models.
  arXiv  Detail & Related papers  (2023-05-30T17:59:26Z)
• Learning to Dub Movies via Hierarchical Prosody Models [167.6465354313349]
  Given a piece of text, a video clip and a reference audio, the movie dubbing (also known as visual voice clone V2C) task aims to generate speeches that match the speaker's emotion presented in the video using the desired speaker voice as reference. We propose a novel movie dubbing architecture to tackle these problems via hierarchical prosody modelling, which bridges the visual information to corresponding speech prosody from three aspects: lip, face, and scene.
  arXiv  Detail & Related papers  (2022-12-08T03:29:04Z)
• Limited Data Emotional Voice Conversion Leveraging Text-to-Speech: Two-stage Sequence-to-Sequence Training [91.95855310211176]
  Emotional voice conversion aims to change the emotional state of an utterance while preserving the linguistic content and speaker identity. We propose a novel 2-stage training strategy for sequence-to-sequence emotional voice conversion with a limited amount of emotional speech data. The proposed framework can perform both spectrum and prosody conversion and achieves significant improvement over the state-of-the-art baselines in both objective and subjective evaluation.
  arXiv  Detail & Related papers  (2021-03-31T04:56:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.