Unified Synthesis of Compositional Speech and Sound from Free-Form Text Prompts

• URL: http://arxiv.org/abs/2605.28063v1
• Date: Wed, 27 May 2026 07:15:35 GMT
• Title: Unified Synthesis of Compositional Speech and Sound from Free-Form Text Prompts
• Abstract summary: We introduce a new task: Free-Form-Text-Prompt-to-Unified-Audio generation.<n>PlanAudio is a unified, autoregressive LLM-based framework.<n>We perform evaluations in the scenarios of speech, sound, and their composites.
• Score: 20.986457042343684
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Audio generation has made significant progress, yet synthesizing unified audio where speech and sounds are naturally composited remains a challenge. Current methods either rely on disjoint pipelines, which fail to capture fine-grained interactions, or require structured inputs and external text rewriting, which limits the flexibility of free-form text prompts. In this paper, we introduce a new task: Free-Form-Text-Prompt-to-Unified-Audio generation, which aims to directly synthesize unified audio containing speech, sound, and their composites from unconstrained natural language. To address this task, we propose PlanAudio, a unified, autoregressive LLM-based framework. First, it simplifies the model architecture by leveraging intrinsic LLM reasoning capability instead of traditional text encoders. Second, it introduces a semantic latent chain-of-thought mechanism, an implicit planning mechanism that bridges high-level semantic understanding and low-level acoustic synthesis. Furthermore, we create PlanAudio-Bench, a specialized benchmark for evaluating composite audio scenarios. We perform evaluations in the scenarios of speech, sound, and their composites. The results demonstrate that PlanAudio generally outperforms the existing pipeline and unified baselines, while staying competitive with models designed for a single scenario. Our analysis further reveals the superiority of semantic latent CoT over other CoT mechanisms and highlights the importance of continuous multi-scenario training curricula.

Related papers

• Reference-Driven Multi-Speaker Audio Scene Generation from In-the-Wild Priors [10.603967327834214]
  Existing multi-speaker dialogue systems bind speakers to utterances through structured supervision.<n>Our method, ScenA, conditions a text-to-audio flow-matching foundation model, pretrained on large-scale in-the-wild data.<n>We evaluate ScenA on the CoVoMix2-Dialogue benchmark, showing that it outperforms existing multi-speaker systems on speaker-binding metrics.
  arXiv  Detail & Related papers  (2026-06-17T17:51:50Z)
• ImmersiveTTS: Environment-Aware Text-to-Speech with Multimodal Diffusion Transformer and Domain-Specific Representation Alignment [52.31881685769569]
  We propose ImmersiveTTS, an environment-aware text-to-speech (TTS) model that generates natural speech seamlessly integrated within environmental contexts.<n>Our model builds on a multimodal diffusion transformer and fuses transcript-aligned speech latent with text-conditioned environmental context.<n> Experimental results show that ImmersiveTTS achieves higher naturalness, intelligibility, and audio fidelity than existing approaches.
  arXiv  Detail & Related papers  (2026-05-29T07:58:54Z)
• UniSonate: A Unified Model for Speech, Music, and Sound Effect Generation with Text Instructions [55.622295453533475]
  We introduce UniSonate, a unified flow-matching framework capable of synthesizing speech, music, and sound effects.<n>We propose a novel dynamic token injection mechanism that projects unstructured environmental sounds into a structured temporal latent space.<n>Experiments demonstrate that UniSonate achieves state-of-the-art performance in instruction-based TTS and TTM.
  arXiv  Detail & Related papers  (2026-04-24T04:26:04Z)
• Borderless Long Speech Synthesis [30.36601404142387]
  We introduce the Borderless Long Speech Synthesis framework for agent-centric, borderless long audio synthesis.<n>Rather than targeting a single narrow task, the system is designed as a unified capability set spanning VoiceDesigner, multi-speaker synthesis, Instruct TTS, and long-form text synthesis.
  arXiv  Detail & Related papers  (2026-03-20T09:37:54Z)
• Covo-Audio Technical Report [61.09708870154148]
  Covo-Audio, a 7B-end LALM, directly processes continuous audio inputs and generates audio outputs within a single unified architecture.<n>Covo-Audio-Chat, a dialogue-oriented variant, demonstrates semantic strong spoken conversational abilities.
  arXiv  Detail & Related papers  (2026-02-10T14:31:11Z)
• MOSS-Speech: Towards True Speech-to-Speech Models Without Text Guidance [66.74042564585942]
  MOSS-Speech is a true speech-to-speech large language model that directly understands and generates speech without relying on text guidance.<n>Our work establishes a new paradigm for expressive and efficient end-to-end speech interaction.
  arXiv  Detail & Related papers  (2025-10-01T04:32:37Z)
• Text-Queried Audio Source Separation via Hierarchical Modeling [53.94434504259829]
  We propose a hierarchical decomposition framework, HSM-TSS, that decouples the task into global-local semantic-guided feature separation and structure-preserving acoustic reconstruction.<n>A Q-Audio architecture is employed to align audio and text modalities, serving as pretrained global-semantic encoders.<n>Our method achieves state-of-the-art separation performance with data-efficient training while maintaining superior semantic consistency with queries in complex auditory scenes.
  arXiv  Detail & Related papers  (2025-05-27T11:00:38Z)
• GOAT-TTS: Expressive and Realistic Speech Generation via A Dual-Branch LLM [42.93855899824886]
  We propose a text-to-speech generation approach optimized via a novel dual-branch ArchiTecture (GOAT-TTS)<n>GOAT-TTS combines a speech encoder and projector to capture continuous acoustic embeddings, enabling bidirectional correlation between paralinguistic features (language, timbre, emotion) and semantic text representations without transcript dependency.<n> Experimental results demonstrate that our GOAT-TTS achieves performance comparable to state-of-the-art TTS models.
  arXiv  Detail & Related papers  (2025-04-15T01:44:56Z)
• CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models [74.80386066714229]
  We present an improved streaming speech synthesis model, CosyVoice 2.<n>Specifically, we introduce finite-scalar quantization to improve codebook utilization of speech tokens.<n>We develop a chunk-aware causal flow matching model to support various synthesis scenarios.
  arXiv  Detail & Related papers  (2024-12-13T12:59:39Z)
• Cross-Utterance Conditioned VAE for Speech Generation [27.5887600344053]
  We present the Cross-Utterance Conditioned Variational Autoencoder speech synthesis (CUC-VAE S2) framework to enhance prosody and ensure natural speech generation. We propose two practical algorithms tailored for distinct speech synthesis applications: CUC-VAE TTS for text-to-speech and CUC-VAE SE for speech editing.
  arXiv  Detail & Related papers  (2023-09-08T06:48:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.