TalkVerse: Democratizing Minute-Long Audio-Driven Video Generation

• URL: http://arxiv.org/abs/2512.14938v1
• Date: Tue, 16 Dec 2025 22:01:08 GMT
• Title: TalkVerse: Democratizing Minute-Long Audio-Driven Video Generation
• Abstract summary: We introduce TalkVerse, a large-scale, open corpus for audio-driven talking video generation.<n>TalkVerse offers 2.3 million high-resolution (720p/1080p) audio-video synchronized clips totaling 6.3k hours.<n>These are curated from over 60k hours of video via a transparent pipeline.
• Score: 45.08904139559434
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce TalkVerse, a large-scale, open corpus for single-person, audio-driven talking video generation designed to enable fair, reproducible comparison across methods. While current state-of-the-art systems rely on closed data or compute-heavy models, TalkVerse offers 2.3 million high-resolution (720p/1080p) audio-video synchronized clips totaling 6.3k hours. These are curated from over 60k hours of video via a transparent pipeline that includes scene-cut detection, aesthetic assessment, strict audio-visual synchronization checks, and comprehensive annotations including 2D skeletons and structured visual/audio-style captions. Leveraging TalkVerse, we present a reproducible 5B DiT baseline built on Wan2.2-5B. By utilizing a video VAE with a high downsampling ratio and a sliding window mechanism with motion-frame context, our model achieves minute-long generation with low drift. It delivers comparable lip-sync and visual quality to the 14B Wan-S2V model but with 10$\times$ lower inference cost. To enhance storytelling in long videos, we integrate an MLLM director to rewrite prompts based on audio and visual cues. Furthermore, our model supports zero-shot video dubbing via controlled latent noise injection. We open-source the dataset, training recipes, and 5B checkpoints to lower barriers for research in audio-driven human video generation. Project Page: https://zhenzhiwang.github.io/talkverse/

Related papers

• Adding Voice Cloning to Text-to-Audio-Video Models with a Single Zero-Initialised Layer [1.5678079269619796]
  We show that a base T2AV model can be turned into a voice-cloning model by adding a single zero-d linear layer on top of its audio backbone.<n>A side product of the architecture is that the audio path can be evaluated without the video path at inference time, yielding a 30x speed-up over the full audio-video diffusion loop.
  arXiv  Detail & Related papers  (2026-08-16T11:40:20Z)
• Speed by Simplicity: A Single-Stream Architecture for Fast Audio-Video Generative Foundation Model [52.79436545460808]
  We present daVinci-MagiHuman, an open-source audio-video generative foundation model for human-centric generation.<n>DaVinci-MagiHuman jointly generates synchronized video and audio using a single-stream Transformer.<n>It supports multilingual spoken generation across Chinese (Mandarin and Cantonese), English, Japanese, Korean, German, and French.
  arXiv  Detail & Related papers  (2026-03-23T13:49:06Z)
• MOVA: Towards Scalable and Synchronized Video-Audio Generation [91.56945636522345]
  We introduce MOVA (MOSS Video and Audio), an open-source model capable of generating high-quality, synchronized audio-visual content.<n>By releasing the model weights and code, we aim to advance research and foster a vibrant community of creators.
  arXiv  Detail & Related papers  (2026-02-09T15:31:54Z)
• ALIVE: Animate Your World with Lifelike Audio-Video Generation [50.693986608051716]
  ALIVE is a generation model that adapts a pretrained Text-to-Video (T2V) model to Sora-style audio-video generation and animation.<n>To support the audio-visual synchronization and reference animation, we augment the popular MMDiT architecture with a joint audio-video branch.<n>ALIVE demonstrates outstanding performance, consistently outperforming open-source models and matching or surpassing state-of-the-art commercial solutions.
  arXiv  Detail & Related papers  (2026-02-09T14:06:03Z)
• JUST-DUB-IT: Video Dubbing via Joint Audio-Visual Diffusion [47.70095297438178]
  We introduce a single-model approach that adapts an audio-video diffusion model for video-to-video dubbing via a lightweight LoRA.<n>We generate multilingual videos with language switches within a single clip, and then inpaint the face and audio in each half to match the language of the other half.<n>We demonstrate that our approach produces high-quality dubbed videos with improved visual fidelity, lip synchronization, and robustness compared to existing dubbing pipelines.
  arXiv  Detail & Related papers  (2026-01-29T18:57:13Z)
• VABench: A Comprehensive Benchmark for Audio-Video Generation [22.00633729850902]
  VABench is a benchmark framework designed to evaluate the capabilities of synchronous audio-video generation.<n>It covers three task types: text-to-audio-video (T2AV), image-to-audio-video (I2AV), and stereo audio-video generation.<n>VABench covers seven major content categories: animals, human sounds, music, environmental sounds, synchronous physical sounds, complex scenes, and virtual worlds.
  arXiv  Detail & Related papers  (2025-12-10T03:57:29Z)
• Democratizing High-Fidelity Co-Speech Gesture Video Generation [25.34048359399198]
  Co-speech gesture video generation aims to synthesize realistic, audio-aligned videos of speakers, complete with synchronized facial expressions and body gestures.<n>We propose a framework that utilizes 2D full-body skeletons as an efficient auxiliary condition to bridge audio signals with visual outputs.<n>We present CSG-405-the first public dataset with 405 hours of high-resolution videos across 71 speech types, annotated with 2D skeletons and diverse speaker demographics.
  arXiv  Detail & Related papers  (2025-07-09T13:02:12Z)
• Aligned Better, Listen Better for Audio-Visual Large Language Models [21.525317311280205]
  Video inherently contains audio, which supplies information to vision.<n>Video large language models (Video-LLMs) can encounter many audio-centric settings.<n>Existing models exhibit deficiencies in exploiting audio information, leading to weak understanding and hallucinations.
  arXiv  Detail & Related papers  (2025-04-02T18:47:09Z)
• Audio-visual training for improved grounding in video-text LLMs [1.9320359360360702]
  We propose a model architecture that handles audio-visual inputs explicitly. We train our model with both audio and visual data from a video instruction-tuning dataset. For better evaluation of audio-visual models, we also release a human-annotated benchmark dataset.
  arXiv  Detail & Related papers  (2024-07-21T03:59:14Z)
• TAVGBench: Benchmarking Text to Audible-Video Generation [44.67058949063691]
  We develop a comprehensive Text to Audible-Video Generation Benchmark (TAVGBench) TAVGBench contains over 1.7 million clips with a total duration of 11.8 thousand hours. We propose an automatic audible annotation pipeline to ensure each video has detailed descriptions for both its audio and video contents.
  arXiv  Detail & Related papers  (2024-04-22T17:36:03Z)
• InternVideo2: Scaling Foundation Models for Multimodal Video Understanding [51.129913789991924]
  InternVideo2 is a new family of video foundation models (FM) that achieve state-of-the-art results in video recognition, video-speech tasks, and video-centric tasks. Our core design is a progressive training approach that unifies the masked video modeling, cross contrastive learning, and prediction token, scaling up to 6B video size.
  arXiv  Detail & Related papers  (2024-03-22T17:57:42Z)
• Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners [69.70590867769408]
  Video and audio content creation serves as the core technique for the movie industry and professional users. Existing diffusion-based methods tackle video and audio generation separately, which hinders the technique transfer from academia to industry. In this work, we aim at filling the gap, with a carefully designed optimization-based framework for cross-visual-audio and joint-visual-audio generation.
  arXiv  Detail & Related papers  (2024-02-27T17:57:04Z)
• ECLIPSE: Efficient Long-range Video Retrieval using Sight and Sound [103.28102473127748]
  We introduce an audiovisual method for long-range text-to-video retrieval. Our approach aims to retrieve minute-long videos that capture complex human actions. Our method is 2.92x faster and 2.34x memory-efficient than long-range video-only approaches.
  arXiv  Detail & Related papers  (2022-04-06T14:43:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.