MOSS-VoiceGenerator: Create Realistic Voices with Natural Language Descriptions

• URL: http://arxiv.org/abs/2603.28086v1
• Date: Mon, 30 Mar 2026 06:40:59 GMT
• Title: MOSS-VoiceGenerator: Create Realistic Voices with Natural Language Descriptions
• Abstract summary: We present MOSS-VoiceGenerator, an open-source instruction-driven voice generation model that creates new timbres directly from natural language prompts.<n>Motivated by the hypothesis that exposure to real-world acoustic variation produces more perceptually natural voices, we train on large-scale expressive speech data sourced from cinematic content.<n>Subjective preference studies demonstrate its superiority in overall performance, instruction-following, and naturalness compared to other voice design models.
• Score: 60.69565588088127
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Voice design from natural language aims to generate speaker timbres directly from free-form textual descriptions, allowing users to create voices tailored to specific roles, personalities, and emotions. Such controllable voice creation benefits a wide range of downstream applications-including storytelling, game dubbing, role-play agents, and conversational assistants, making it a significant task for modern Text-to-Speech models. However, existing models are largely trained on carefully recorded studio data, which produces speech that is clean and well-articulated, yet lacks the lived-in qualities of real human voices. To address these limitations, we present MOSS-VoiceGenerator, an open-source instruction-driven voice generation model that creates new timbres directly from natural language prompts. Motivated by the hypothesis that exposure to real-world acoustic variation produces more perceptually natural voices, we train on large-scale expressive speech data sourced from cinematic content. Subjective preference studies demonstrate its superiority in overall performance, instruction-following, and naturalness compared to other voice design models.

Related papers

• VoiceDesigner: Text-to-Voice Generation and Editing via Unified Diffusion Modeling and Data Augmentation [33.076169399416855]
  Existing systems struggle to generate a diverse range of voices, spanning real-world human speakers and fictional characters.<n>We propose VoiceDesigner, a unified framework for text-to-voice generation and editing that supports diverse and controllable voice design.
  arXiv  Detail & Related papers  (2026-08-12T18:04:18Z)
• NaturalVoices: A Large-Scale, Spontaneous and Emotional Podcast Dataset for Voice Conversion [25.896735200803537]
  NaturalVoices (NV) is the first large-scale spontaneous podcast dataset specifically designed for emotion-aware voice conversion.<n>It comprises 5,049 hours of spontaneous podcast recordings with automatic annotations for emotion (categorical and attribute-based), speech quality, transcripts, speaker identity, and sound events.<n>The dataset captures expressive emotional variation across thousands of speakers, diverse topics, and natural speaking styles.
  arXiv  Detail & Related papers  (2025-10-31T21:00:14Z)
• BatonVoice: An Operationalist Framework for Enhancing Controllable Speech Synthesis with Linguistic Intelligence from LLMs [84.59993864748195]
  We propose a new paradigm inspired by operationalism'' that decouples instruction understanding from speech generation.<n>We introduce BatonVoice, a framework where an LLM acts as a conductor'', understanding user instructions.<n>A separate TTS model, the orchestra'', then generates the speech from these features.
  arXiv  Detail & Related papers  (2025-09-30T16:52:14Z)
• Revival with Voice: Multi-modal Controllable Text-to-Speech Synthesis [52.25128289155576]
  This paper explores multi-modal controllable Text-to-Speech Synthesis (TTS) where the voice can be generated from face image.<n>We aim to mitigate the following three challenges in face-driven TTS systems.<n> Experimental results validate the proposed model's effectiveness in face-driven voice synthesis.
  arXiv  Detail & Related papers  (2025-05-25T04:43:17Z)
• LoRP-TTS: Low-Rank Personalized Text-To-Speech [0.0]
  Speech synthesis models convert written text into natural-sounding audio.<n>Low-Rank Adaptation (LoRA) allows us to successfully use even single recordings of spontaneous speech in noisy environments as prompts.
  arXiv  Detail & Related papers  (2025-02-11T14:00:12Z)
• A Unified Model For Voice and Accent Conversion In Speech and Singing using Self-Supervised Learning and Feature Extraction [0.0]
  The paper presents a new voice conversion model capable of transforming both speaking and singing voices.<n>One of the model's standout features is its ability to perform accent conversion on hybrid voice samples.
  arXiv  Detail & Related papers  (2024-12-11T11:47:39Z)
• FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs [63.8261207950923]
  FunAudioLLM is a model family designed to enhance natural voice interactions between humans and large language models (LLMs) At its core are two innovative models: SenseVoice, which handles multilingual speech recognition, emotion recognition, and audio event detection; and CosyVoice, which facilitates natural speech generation with control over multiple languages, timbre, speaking style, and speaker identity. The models related to SenseVoice and CosyVoice have been open-sourced on Modelscope and Huggingface, along with the corresponding training, inference, and fine-tuning codes released on GitHub.
  arXiv  Detail & Related papers  (2024-07-04T16:49:02Z)
• Prompt-Singer: Controllable Singing-Voice-Synthesis with Natural Language Prompt [50.25271407721519]
  We propose Prompt-Singer, the first SVS method that enables attribute controlling on singer gender, vocal range and volume with natural language.<n>We adopt a model architecture based on a decoder-only transformer with a multi-scale hierarchy, and design a range-melody decoupled pitch representation.<n>Experiments show that our model achieves favorable controlling ability and audio quality.
  arXiv  Detail & Related papers  (2024-03-18T13:39:05Z)
• Natural language guidance of high-fidelity text-to-speech with synthetic annotations [13.642358232817342]
  We propose a scalable method for labeling various aspects of speaker identity, style, and recording conditions. We then apply this method to a 45k hour dataset, which we use to train a speech language model. Our results demonstrate high-fidelity speech generation in a diverse range of accents, prosodic styles, channel conditions, and acoustic conditions.
  arXiv  Detail & Related papers  (2024-02-02T21:29:34Z)
• uSee: Unified Speech Enhancement and Editing with Conditional Diffusion Models [57.71199494492223]
  We propose a Unified Speech Enhancement and Editing (uSee) model with conditional diffusion models to handle various tasks at the same time in a generative manner. Our experiments show that our proposed uSee model can achieve superior performance in both speech denoising and dereverberation compared to other related generative speech enhancement models.
  arXiv  Detail & Related papers  (2023-10-02T04:36:39Z)
• Zero-shot personalized lip-to-speech synthesis with face image based voice control [41.17483247506426]
  Lip-to-Speech (Lip2Speech) synthesis, which predicts corresponding speech from talking face images, has witnessed significant progress with various models and training strategies. We propose a zero-shot personalized Lip2Speech synthesis method, in which face images control speaker identities.
  arXiv  Detail & Related papers  (2023-05-09T02:37:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.