UniVoice: A Unified Model for Speech and Singing Voice Generation

• URL: http://arxiv.org/abs/2606.05852v1
• Date: Thu, 04 Jun 2026 08:27:17 GMT
• Title: UniVoice: A Unified Model for Speech and Singing Voice Generation
• Abstract summary: We present UniVoice, a unified speech and singing voice generation framework based on conditional flow matching.<n>For singing, the melody condition is represented by MIDI note sequences; for speech, it is replaced with a learned null melody token.<n>UniVoice achieves a speech PER of 5.26%, comparable to dedicated TTS systems such as F5-TTS (5.21%) and CosyVoice3 (5.30%)
• Score: 11.813888122703302
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Text-to-speech (TTS) and singing voice synthesis (SVS) both aim to generate human vocal audio from symbolic inputs, but they impose different requirements on the generation process. Speech generation relies on flexible, language-driven prosody, whereas singing generation requires explicit melody control and accurate rhythmic alignment. This mismatch makes it challenging to train a single model that can generate both natural speech and controllable singing, since melody-related conditions should strongly constrain singing but should not restrict speech prosody. We present UniVoice, a unified speech and singing voice generation framework based on conditional flow matching. Instead of using a single undifferentiated conditioning representation, UniVoice factorizes the condition into content, melody, and timbre, which are encoded by modality-appropriate encoders and consumed by a shared Diffusion Transformer (DiT) backbone. For singing, the melody condition is represented by MIDI note sequences; for speech, it is replaced with a learned null melody token, allowing the model to infer prosody from linguistic and acoustic context. This design preserves explicit melody control for singing while avoiding the need to impose melody constraints on speech. We further analyze the null melody token as an approximation to melody marginalization in the conditional flow. Trained on 30k hours of speech and 35k hours of singing data, UniVoice achieves a speech PER of 5.26\%, comparable to dedicated TTS systems such as F5-TTS (5.21\%) and CosyVoice3 (5.30\%). On singing generation, UniVoice achieves a PER of 16.22\%, outperforming the unified baseline Vevo1.5 (24.72\%).

Related papers

• CookVoice: Unified Framework for Style Controllable Multi-Modal Human Voice Generation [18.80525296281322]
  CookVoice is a unified framework for multimodal, multi-style, and multi-task human voice generation.<n>It decomposes the human voice into three key factors: content, prosody, and style.<n>CookVoice supports text-to-speech, text-to-singing voice, style-controllable generation, voice mimicry, voice conversion, and voice editing.
  arXiv  Detail & Related papers  (2026-08-12T02:58:45Z)
• X-Voice: Enabling Everyone to Speak 30 Languages via Zero-Shot Cross-Lingual Voice Cloning [62.740127542449166]
  X-Voice is a multilingual zero-shot voice cloning model that clones arbitrary voices and enables everyone to speak 30 languages.<n>X-Voice is trained on a 420K-hour multilingual corpus using the International Phonetic Alphabet (IPA) as a unified representation.
  arXiv  Detail & Related papers  (2026-05-07T02:57:53Z)
• VocalParse: Towards Unified and Scalable Singing Voice Transcription with Large Audio Language Models [45.70355425430011]
  We present VocalParse, a unified singing voice transcription (SVT) model built upon a Large Audio Language Model (LALM)<n>Our novel contribution is to introduce an interleaved prompting formulation that jointly models lyrics, melody, and word-note correspondence, yielding a generated sequence that directly maps to a structured musical score.<n> Experiments demonstrate that VocalParse achieves state-of-the-art SVT performance on multiple singing datasets.
  arXiv  Detail & Related papers  (2026-05-06T08:03:31Z)
• Vevo2: Bridging Controllable Speech and Singing Voice Generation via Unified Prosody Learning [19.2319862239494]
  This paper introduces Vevo2, a unified framework for controllable speech and singing voice generation.<n>Vevo2 consists of an auto-regressive (AR) content-style modeling stage, which aims to enable controllability over text, prosody, and style.<n> Experimental results show that the unified modeling in Vevo2 brings mutual benefits to both speech and singing voice generation.
  arXiv  Detail & Related papers  (2025-08-22T12:20:11Z)
• TCSinger: Zero-Shot Singing Voice Synthesis with Style Transfer and Multi-Level Style Control [58.96445085236971]
  Zero-shot singing voice synthesis (SVS) with style transfer and style control aims to generate high-quality singing voices with unseen timbres and styles.<n>We introduce TCSinger, the first zero-shot SVS model for style transfer across cross-lingual speech and singing styles.
  arXiv  Detail & Related papers  (2024-09-24T11:18:09Z)
• S2Cap: A Benchmark and a Baseline for Singing Style Captioning [12.515874333424929]
  We present S2Cap, a dataset of singing voices with detailed descriptions covering diverse vocal, acoustic, and demographic characteristics.<n>Using this dataset, we develop an efficient and straightforward baseline algorithm for singing style captioning.
  arXiv  Detail & Related papers  (2024-09-15T21:19:24Z)
• Accompanied Singing Voice Synthesis with Fully Text-controlled Melody [61.147446955297625]
  Text-to-song (TTSong) is a music generation task that synthesizes accompanied singing voices. We present MelodyLM, the first TTSong model that generates high-quality song pieces with fully text-controlled melodies.
  arXiv  Detail & Related papers  (2024-07-02T08:23:38Z)
• Prompt-Singer: Controllable Singing-Voice-Synthesis with Natural Language Prompt [50.25271407721519]
  We propose Prompt-Singer, the first SVS method that enables attribute controlling on singer gender, vocal range and volume with natural language.<n>We adopt a model architecture based on a decoder-only transformer with a multi-scale hierarchy, and design a range-melody decoupled pitch representation.<n>Experiments show that our model achieves favorable controlling ability and audio quality.
  arXiv  Detail & Related papers  (2024-03-18T13:39:05Z)
• StyleSinger: Style Transfer for Out-of-Domain Singing Voice Synthesis [63.18764165357298]
  Style transfer for out-of-domain singing voice synthesis (SVS) focuses on generating high-quality singing voices with unseen styles.<n>StyleSinger is the first singing voice synthesis model for zero-shot style transfer of out-of-domain reference singing voice samples.<n>Our evaluations in zero-shot style transfer undeniably establish that StyleSinger outperforms baseline models in both audio quality and similarity to the reference singing voice samples.
  arXiv  Detail & Related papers  (2023-12-17T15:26:16Z)
• Make-A-Voice: Unified Voice Synthesis With Discrete Representation [77.3998611565557]
  Make-A-Voice is a unified framework for synthesizing and manipulating voice signals from discrete representations. We show that Make-A-Voice exhibits superior audio quality and style similarity compared with competitive baseline models.
  arXiv  Detail & Related papers  (2023-05-30T17:59:26Z)
• A Melody-Unsupervision Model for Singing Voice Synthesis [9.137554315375919]
  We propose a melody-unsupervision model that requires only audio-and-lyrics pairs without temporal alignment in training time. We show that the proposed model is capable of being trained with speech audio and text labels but can generate singing voice in inference time.
  arXiv  Detail & Related papers  (2021-10-13T07:42:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.