VoiceDesigner: Text-to-Voice Generation and Editing via Unified Diffusion Modeling and Data Augmentation

• URL: http://arxiv.org/abs/2608.13613v1
• Date: Wed, 12 Aug 2026 18:04:18 GMT
• Title: VoiceDesigner: Text-to-Voice Generation and Editing via Unified Diffusion Modeling and Data Augmentation
• Abstract summary: Existing systems struggle to generate a diverse range of voices, spanning real-world human speakers and fictional characters.<n>We propose VoiceDesigner, a unified framework for text-to-voice generation and editing that supports diverse and controllable voice design.
• Score: 33.076169399416855
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent breakthroughs in generative models have made text-to-voice generation (TTV) possible, enabling the synthesis of speech directly from textual voice descriptions. However, existing systems face two key challenges. First, they struggle to generate a diverse range of voices, spanning real-world human speakers and fictional characters. Second, they lack robust and flexible voice editing capabilities, such as voice cloning and the ability to modify attributes like emotion and tone. In this paper, we propose VoiceDesigner, a unified framework for text-to-voice generation and editing that supports diverse and controllable voice design. To tackle the above challenges, we propose solutions from two perspectives. First, we develop a hybrid data pipeline that leverages digital signal processing techniques and speech generation models to construct a diverse voice dataset covering both real-world and fictional voices. Second, we introduce a diffusion transformer with architectural improvements to better handle complex conditioning and enhance multi-task performance, enabling unified voice generation and editing. Through subjective and objective evaluations, VoiceDesigner achieves superior prompt alignment with both voice descriptions and editing instructions, while maintaining competitive perceptual quality and voice usability compared to state-of-the-art TTV models.

Related papers

• CookVoice: Unified Framework for Style Controllable Multi-Modal Human Voice Generation [18.80525296281322]
  CookVoice is a unified framework for multimodal, multi-style, and multi-task human voice generation.<n>It decomposes the human voice into three key factors: content, prosody, and style.<n>CookVoice supports text-to-speech, text-to-singing voice, style-controllable generation, voice mimicry, voice conversion, and voice editing.
  arXiv  Detail & Related papers  (2026-08-12T02:58:45Z)
• MOSS-VoiceGenerator: Create Realistic Voices with Natural Language Descriptions [60.69565588088127]
  We present MOSS-VoiceGenerator, an open-source instruction-driven voice generation model that creates new timbres directly from natural language prompts.<n>Motivated by the hypothesis that exposure to real-world acoustic variation produces more perceptually natural voices, we train on large-scale expressive speech data sourced from cinematic content.<n>Subjective preference studies demonstrate its superiority in overall performance, instruction-following, and naturalness compared to other voice design models.
  arXiv  Detail & Related papers  (2026-03-30T06:40:59Z)
• Revival with Voice: Multi-modal Controllable Text-to-Speech Synthesis [52.25128289155576]
  This paper explores multi-modal controllable Text-to-Speech Synthesis (TTS) where the voice can be generated from face image.<n>We aim to mitigate the following three challenges in face-driven TTS systems.<n> Experimental results validate the proposed model's effectiveness in face-driven voice synthesis.
  arXiv  Detail & Related papers  (2025-05-25T04:43:17Z)
• A Unified Model For Voice and Accent Conversion In Speech and Singing using Self-Supervised Learning and Feature Extraction [0.0]
  The paper presents a new voice conversion model capable of transforming both speaking and singing voices.<n>One of the model's standout features is its ability to perform accent conversion on hybrid voice samples.
  arXiv  Detail & Related papers  (2024-12-11T11:47:39Z)
• Prompt-Singer: Controllable Singing-Voice-Synthesis with Natural Language Prompt [50.25271407721519]
  We propose Prompt-Singer, the first SVS method that enables attribute controlling on singer gender, vocal range and volume with natural language.<n>We adopt a model architecture based on a decoder-only transformer with a multi-scale hierarchy, and design a range-melody decoupled pitch representation.<n>Experiments show that our model achieves favorable controlling ability and audio quality.
  arXiv  Detail & Related papers  (2024-03-18T13:39:05Z)
• Make-A-Voice: Unified Voice Synthesis With Discrete Representation [77.3998611565557]
  Make-A-Voice is a unified framework for synthesizing and manipulating voice signals from discrete representations. We show that Make-A-Voice exhibits superior audio quality and style similarity compared with competitive baseline models.
  arXiv  Detail & Related papers  (2023-05-30T17:59:26Z)
• Limited Data Emotional Voice Conversion Leveraging Text-to-Speech: Two-stage Sequence-to-Sequence Training [91.95855310211176]
  Emotional voice conversion aims to change the emotional state of an utterance while preserving the linguistic content and speaker identity. We propose a novel 2-stage training strategy for sequence-to-sequence emotional voice conversion with a limited amount of emotional speech data. The proposed framework can perform both spectrum and prosody conversion and achieves significant improvement over the state-of-the-art baselines in both objective and subjective evaluation.
  arXiv  Detail & Related papers  (2021-03-31T04:56:14Z)
• VoiceCoach: Interactive Evidence-based Training for Voice Modulation Skills in Public Speaking [55.366941476863644]
  The modulation of voice properties, such as pitch, volume, and speed, is crucial for delivering a successful public speech. We present VoiceCoach, an interactive evidence-based approach to facilitate the effective training of voice modulation skills.
  arXiv  Detail & Related papers  (2020-01-22T04:52:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.