AudioX-Turbo: A Unified Framework for Efficient Anything-to-Audio Generation

• URL: http://arxiv.org/abs/2606.12555v1
• Date: Wed, 10 Jun 2026 18:06:27 GMT
• Title: AudioX-Turbo: A Unified Framework for Efficient Anything-to-Audio Generation
• Abstract summary: We propose AudioX-Turbo, a framework for anything-to-audio generation that integrates varied multimodal conditions.<n>To support the training of AudioX-Turbo, we construct a large-scale, high-quality dataset, IF-caps-Pro, comprising approximately 9.2M samples.<n>We benchmark AudioX-Turbo across a wide range of tasks, finding that our model achieves superior performance, especially on text-to-audio and text-to-music generation.
• Score: 111.61779149559645
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Audio and music generation based on flexible multimodal control signals is a widely applicable topic, with the following key challenges: 1) a unified multimodal modeling framework, 2) large-scale, high-quality training data, and 3) the prohibitive inference cost of multi-step diffusion sampling. As such, we propose AudioX-Turbo, a unified and efficient framework for anything-to-audio generation that integrates varied multimodal conditions (i.e., text, video, and audio signals) in this work. AudioX-Turbo follows a teacher-student paradigm. The teacher AudioX-Base is built on a Multimodal Diffusion Transformer with a Multimodal Adaptive Fusion module that aligns diverse multimodal inputs for high-fidelity synthesis, and is then distilled into the few-step student AudioX-Turbo via Distribution Matching Distillation adapted to flow matching, complemented by a diffusion-based discriminator for high-quality few-step generation. To support the training of AudioX-Turbo, we construct a large-scale, high-quality dataset, IF-caps-Pro, comprising approximately 9.2M samples curated through a two-stage data collection and annotation pipeline. We benchmark AudioX-Turbo across a wide range of tasks, finding that our model achieves superior performance, especially on text-to-audio and text-to-music generation, while operating at only 4 sampling steps and requiring approximately 25x fewer function evaluations (NFE) than multi-step baselines. These results demonstrate that our method is capable of audio generation under flexible multimodal control, showing efficient and powerful instruction-following capabilities. The code and datasets will be available at https://zeyuet.github.io/AudioX-Turbo/.

Related papers

• DreamX-Creator: Democratizing Native Audio-Video Generation at 2K Resolution [16.570672947240855]
  We present DreamX-Creator 1.0, a compact native joint audio-video generation system centered on a 7B generator.<n>A unified Audio-Video Data System constructs and filters temporally coherent clips, produces structured multimodal annotations, and organizes clips into capability-oriented data pools.<n>For high-resolution output, our Autoregressive 1-Step 2K Refinement pipeline adapts a bidirectional multi-step teacher into an autoregressive multi-step refiner.
  arXiv  Detail & Related papers  (2026-08-31T17:11:05Z)
• Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editing [63.573256490583724]
  Audio- Omni is the first end-to-end framework to unify generation and editing across general sound, music, and speech domains.<n>Our architecture synergizes a frozen Multimodal Large Language Model for high-level reasoning with a trainable Diffusion Transformer for high-fidelity synthesis.<n>AudioEdit is a large-scale dataset comprising over one million meticulously curated editing pairs.
  arXiv  Detail & Related papers  (2026-04-12T16:08:20Z)
• Training-Free Multimodal Guidance for Video to Audio Generation [22.64037676707457]
  Video-to-audio (V2A) generation aims to synthesize realistic and semantically aligned audio from silent videos.<n>Existing approaches either require costly joint training on large-scale paired datasets or rely on pairwise similarities.<n>We propose a novel training-free multimodal guidance mechanism for V2A diffusion that leverages the volume spanned by the modality embeddings to enforce unified alignment.
  arXiv  Detail & Related papers  (2025-09-29T10:00:36Z)
• AlignDiT: Multimodal Aligned Diffusion Transformer for Synchronized Speech Generation [65.06374691172061]
  multimodal-to-speech task has gained increasing attention due to its wide range of applications, such as film production, dubbing, and virtual avatars.<n>Existing methods still suffer from limitations in speech intelligibility, audio-video synchronization, speech naturalness, and voice similarity to the reference speaker.<n>We propose AlignDiT, a multimodal Aligned Diffusion Transformer that generates accurate, synchronized, and natural-sounding speech from aligned multimodal inputs.
  arXiv  Detail & Related papers  (2025-04-29T10:56:24Z)
• AudioX: Diffusion Transformer for Anything-to-Audio Generation [72.84633243365093]
  AudioX is a unified Diffusion Transformer model for Anything-to-Audio and Music Generation.<n>It can generate both general audio and music with high quality, while offering flexible natural language control.<n>To address data scarcity, we curate two datasets: vggsound-caps with 190K audio captions based on the VGGSound dataset, and V2M-caps with 6 million music captions derived from the V2M dataset.
  arXiv  Detail & Related papers  (2025-03-13T16:30:59Z)
• UniForm: A Unified Multi-Task Diffusion Transformer for Audio-Video Generation [44.21422404659117]
  UniForm is a unified multi-task diffusion transformer that generates both audio and visual modalities in a shared latent space.<n>By using a unified denoising network, UniForm captures the inherent correlations between sound and vision.<n>By leveraging large language models and a large-scale text-audio-video combined dataset, UniForm achieves greater generative diversity than prior approaches.
  arXiv  Detail & Related papers  (2025-02-06T09:18:30Z)
• Beyond Single-Audio: Advancing Multi-Audio Processing in Audio Large Language Models [56.776580717999806]
  Real-world applications often involve processing multiple audio streams simultaneously. We propose the first multi-audio evaluation benchmark that consists of 20 datasets from 11 multi-audio tasks. We propose a novel multi-audio-LLM (MALLM) to capture audio context among multiple similar audios.
  arXiv  Detail & Related papers  (2024-09-27T12:06:53Z)
• MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation [70.74377373885645]
  We propose the first joint audio-video generation framework that brings engaging watching and listening experiences simultaneously. MM-Diffusion consists of a sequential multi-modal U-Net for a joint denoising process by design. Experiments show superior results in unconditional audio-video generation, and zero-shot conditional tasks.
  arXiv  Detail & Related papers  (2022-12-19T14:11:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.