CookVoice: Unified Framework for Style Controllable Multi-Modal Human Voice Generation

• URL: http://arxiv.org/abs/2608.11590v2
• Date: Thu, 13 Aug 2026 01:32:48 GMT
• Title: CookVoice: Unified Framework for Style Controllable Multi-Modal Human Voice Generation
• Abstract summary: CookVoice is a unified framework for multimodal, multi-style, and multi-task human voice generation.<n>It decomposes the human voice into three key factors: content, prosody, and style.<n>CookVoice supports text-to-speech, text-to-singing voice, style-controllable generation, voice mimicry, voice conversion, and voice editing.
• Score: 18.80525296281322
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Human voice generation has made rapid progress in speech generation, singing voice generation, voice cloning, and voice editing. However, most existing systems are designed for specific tasks and often rely on task-dependent architectures, control signals, or autoregressive decoding, limiting fine-grained controllability and inference efficiency. In this paper, we propose CookVoice, a unified framework for multimodal, multi-style, and multi-task human voice generation. CookVoice decomposes the human voice into three key factors: content, prosody, and style, enabling both speech and singing voice generation within a unified model. To achieve precise and flexible controllability, we design a flexible alignment strategy that maps text, style, and prosody control signals onto the frame-level of spectrogram. This design allows CookVoice to support a wide range of tasks, including text-to-speech, text-to-singing voice, style-controllable generation, voice mimicry, voice conversion, and voice editing. Experimental results show that CookVoice achieves generation quality comparable to existing Text-to-Speech and text-to-singing voice baselines, while providing stronger style and prosody controllability. Moreover, CookVoice achieves comparable performance to large-scale baselines with only 43.51 million parameters and efficient inference using as few as 4 ODE steps, making it a practical solution for real-world human voice generation applications. Demo page is available at https://haoweilou.github.io/CookVoice/.

Related papers

• VoiceDesigner: Text-to-Voice Generation and Editing via Unified Diffusion Modeling and Data Augmentation [33.076169399416855]
  Existing systems struggle to generate a diverse range of voices, spanning real-world human speakers and fictional characters.<n>We propose VoiceDesigner, a unified framework for text-to-voice generation and editing that supports diverse and controllable voice design.
  arXiv  Detail & Related papers  (2026-08-12T18:04:18Z)
• UniVoice: A Unified Model for Speech and Singing Voice Generation [11.813888122703302]
  We present UniVoice, a unified speech and singing voice generation framework based on conditional flow matching.<n>For singing, the melody condition is represented by MIDI note sequences; for speech, it is replaced with a learned null melody token.<n>UniVoice achieves a speech PER of 5.26%, comparable to dedicated TTS systems such as F5-TTS (5.21%) and CosyVoice3 (5.30%)
  arXiv  Detail & Related papers  (2026-06-04T08:27:17Z)
• X-Voice: Enabling Everyone to Speak 30 Languages via Zero-Shot Cross-Lingual Voice Cloning [62.740127542449166]
  X-Voice is a multilingual zero-shot voice cloning model that clones arbitrary voices and enables everyone to speak 30 languages.<n>X-Voice is trained on a 420K-hour multilingual corpus using the International Phonetic Alphabet (IPA) as a unified representation.
  arXiv  Detail & Related papers  (2026-05-07T02:57:53Z)
• MOSS-VoiceGenerator: Create Realistic Voices with Natural Language Descriptions [60.69565588088127]
  We present MOSS-VoiceGenerator, an open-source instruction-driven voice generation model that creates new timbres directly from natural language prompts.<n>Motivated by the hypothesis that exposure to real-world acoustic variation produces more perceptually natural voices, we train on large-scale expressive speech data sourced from cinematic content.<n>Subjective preference studies demonstrate its superiority in overall performance, instruction-following, and naturalness compared to other voice design models.
  arXiv  Detail & Related papers  (2026-03-30T06:40:59Z)
• Revival with Voice: Multi-modal Controllable Text-to-Speech Synthesis [52.25128289155576]
  This paper explores multi-modal controllable Text-to-Speech Synthesis (TTS) where the voice can be generated from face image.<n>We aim to mitigate the following three challenges in face-driven TTS systems.<n> Experimental results validate the proposed model's effectiveness in face-driven voice synthesis.
  arXiv  Detail & Related papers  (2025-05-25T04:43:17Z)
• FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs [63.8261207950923]
  FunAudioLLM is a model family designed to enhance natural voice interactions between humans and large language models (LLMs) At its core are two innovative models: SenseVoice, which handles multilingual speech recognition, emotion recognition, and audio event detection; and CosyVoice, which facilitates natural speech generation with control over multiple languages, timbre, speaking style, and speaker identity. The models related to SenseVoice and CosyVoice have been open-sourced on Modelscope and Huggingface, along with the corresponding training, inference, and fine-tuning codes released on GitHub.
  arXiv  Detail & Related papers  (2024-07-04T16:49:02Z)
• Prompt-Singer: Controllable Singing-Voice-Synthesis with Natural Language Prompt [50.25271407721519]
  We propose Prompt-Singer, the first SVS method that enables attribute controlling on singer gender, vocal range and volume with natural language.<n>We adopt a model architecture based on a decoder-only transformer with a multi-scale hierarchy, and design a range-melody decoupled pitch representation.<n>Experiments show that our model achieves favorable controlling ability and audio quality.
  arXiv  Detail & Related papers  (2024-03-18T13:39:05Z)
• OpenVoice: Versatile Instant Voice Cloning [22.217256641284106]
  We introduce OpenVoice, a versatile voice cloning approach. It requires only a short audio clip from the reference speaker to replicate their voice and generate speech in multiple languages. OpenVoice has been used by more than 2M users worldwide as the voice engine of MyShell.ai.
  arXiv  Detail & Related papers  (2023-12-03T18:41:54Z)
• Make-A-Voice: Unified Voice Synthesis With Discrete Representation [77.3998611565557]
  Make-A-Voice is a unified framework for synthesizing and manipulating voice signals from discrete representations. We show that Make-A-Voice exhibits superior audio quality and style similarity compared with competitive baseline models.
  arXiv  Detail & Related papers  (2023-05-30T17:59:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.