Speed by Simplicity: A Single-Stream Architecture for Fast Audio-Video Generative Foundation Model
- URL: http://arxiv.org/abs/2603.21986v1
- Date: Mon, 23 Mar 2026 13:49:06 GMT
- Title: Speed by Simplicity: A Single-Stream Architecture for Fast Audio-Video Generative Foundation Model
- Abstract summary: We present daVinci-MagiHuman, an open-source audio-video generative foundation model for human-centric generation.<n>DaVinci-MagiHuman jointly generates synchronized video and audio using a single-stream Transformer.<n>It supports multilingual spoken generation across Chinese (Mandarin and Cantonese), English, Japanese, Korean, German, and French.
- Score: 52.79436545460808
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: We present daVinci-MagiHuman, an open-source audio-video generative foundation model for human-centric generation. daVinci-MagiHuman jointly generates synchronized video and audio using a single-stream Transformer that processes text, video, and audio within a unified token sequence via self-attention only. This single-stream design avoids the complexity of multi-stream or cross-attention architectures while remaining easy to optimize with standard training and inference infrastructure. The model is particularly strong in human-centric scenarios, producing expressive facial performance, natural speech-expression coordination, realistic body motion, and precise audio-video synchronization. It supports multilingual spoken generation across Chinese (Mandarin and Cantonese), English, Japanese, Korean, German, and French. For efficient inference, we combine the single-stream backbone with model distillation, latent-space super-resolution, and a Turbo VAE decoder, enabling generation of a 5-second 256p video in 2 seconds on a single H100 GPU. In automatic evaluation, daVinci-MagiHuman achieves the highest visual quality and text alignment among leading open models, along with the lowest word error rate (14.60%) for speech intelligibility. In pairwise human evaluation, it achieves win rates of 80.0% against Ovi 1.1 and 60.9% against LTX 2.3 over 2000 comparisons. We open-source the complete model stack, including the base model, the distilled model, the super-resolution model, and the inference codebase.
Related papers
- TontaubeV1: Streaming Text-to-Speech with Hierarchical Codec Modeling and Bounded Context [0.04583541422554719]
TontaubeV1 is a model that preserves natural prosody while enabling streaming from a single consumer GPU.<n>Our design assumes that prosodic structure is largely established when the semantic stream is generated.<n>The model accepts up to one minute of reference audio for voice conditioning and is designed primarily for English and German.
arXiv Detail & Related papers (2026-09-08T13:03:46Z) - JavisDiT++: Unified Modeling and Optimization for Joint Audio-Video Generation [112.614973927778]
Joint audio-video generation (JAVG) produces synchronized and semantically aligned sound and vision from textual descriptions.<n>This paper presents JavisDiT++, a framework for unified modeling and optimization of JAVG.<n>Our model achieves state-of-the-art performance merely with around 1M public training entries.
arXiv Detail & Related papers (2026-02-22T12:44:28Z) - ALIVE: Animate Your World with Lifelike Audio-Video Generation [50.693986608051716]
ALIVE is a generation model that adapts a pretrained Text-to-Video (T2V) model to Sora-style audio-video generation and animation.<n>To support the audio-visual synchronization and reference animation, we augment the popular MMDiT architecture with a joint audio-video branch.<n>ALIVE demonstrates outstanding performance, consistently outperforming open-source models and matching or surpassing state-of-the-art commercial solutions.
arXiv Detail & Related papers (2026-02-09T14:06:03Z) - JUST-DUB-IT: Video Dubbing via Joint Audio-Visual Diffusion [47.70095297438178]
We introduce a single-model approach that adapts an audio-video diffusion model for video-to-video dubbing via a lightweight LoRA.<n>We generate multilingual videos with language switches within a single clip, and then inpaint the face and audio in each half to match the language of the other half.<n>We demonstrate that our approach produces high-quality dubbed videos with improved visual fidelity, lip synchronization, and robustness compared to existing dubbing pipelines.
arXiv Detail & Related papers (2026-01-29T18:57:13Z) - SoundReactor: Frame-level Online Video-to-Audio Generation [39.113214321291586]
Video-to-Audio generation models operate offline, assuming an entire video sequence or chunks of frames are available beforehand.<n>We introduce the novel task of frame-level online V2A generation, where a model autoregressively generates audio from video without access to future video frames.<n>SoundReactor is the first simple yet effective framework explicitly tailored for this task.
arXiv Detail & Related papers (2025-10-02T15:18:00Z) - Qwen3-Omni Technical Report [105.11829337290249]
We present Qwen3- Omni, a single multimodal model that maintains state-of-the-art performance across text, image, audio, and video.<n>Qwen3- Omni matches the performance of same-sized single-modal models within the Qwen series and excels particularly on audio tasks.<n>It supports text interaction in 119 languages, speech understanding in 19 languages, and speech generation in 10 languages.
arXiv Detail & Related papers (2025-09-22T13:26:24Z) - VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Model [84.25283710008785]
VITA-Audio is an end-to-end large speech model with fast audio-text token generation.<n>MCTP module efficiently generates multiple audio tokens within a single model forward pass.<n>Four-stage progressive training strategy is explored to achieve model acceleration with minimal loss of speech quality.
arXiv Detail & Related papers (2025-05-06T17:59:53Z) - xGen-VideoSyn-1: High-fidelity Text-to-Video Synthesis with Compressed Representations [120.52120919834988]
xGen-SynVideo-1 is a text-to-video (T2V) generation model capable of producing realistic scenes from textual descriptions.
VidVAE compresses video data both spatially and temporally, significantly reducing the length of visual tokens.
DiT model incorporates spatial and temporal self-attention layers, enabling robust generalization across different timeframes and aspect ratios.
arXiv Detail & Related papers (2024-08-22T17:55:22Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.