Vorch-Streamer: Extending Human Audio-Visual Generation to Real-Time Long-Form Streaming

• URL: http://arxiv.org/abs/2608.05663v2
• Date: Fri, 07 Aug 2026 03:30:22 GMT
• Title: Vorch-Streamer: Extending Human Audio-Visual Generation to Real-Time Long-Form Streaming
• Abstract summary: Vorch-Streamer is a post-training framework that enables real-time long-form Text-to-Audio-Video (T2AV) streaming.<n>We construct a synthetic corpus of 80K avatar clips spanning 12-21 seconds and first train a causal generator with mixed Teacher Forcing and Diffusion Forcing.<n>With bounded causal context and four-step denoising, Vorch-Streamer jointly generates audio and video from text at 27.12 FPS, exceeding the 24-FPS real-time playback rate.
• Score: 24.73791931334844
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Real-time long-form avatar audio-video generation requires causal, continuous synthesis while maintaining audiovisual synchronization and visual consistency. Adapting a pretrained bidirectional model to this setting presents two key dilemmas. First, autoregressively reusing generated blocks as context creates exposure bias, causing errors and visual drift to accumulate over long rollouts. Second, a global speech utterance does not indicates a causal generator which portion should be spoken next when only limited local audio-video context is available. We present Vorch-Streamer, a post-training framework that addresses these challenges and enables real-time long-form Text-to-Audio-Video (T2AV) streaming. We construct a synthetic corpus of 80K avatar clips spanning 12-21 seconds and first train a causal generator with mixed Teacher Forcing and Diffusion Forcing. We then apply long-horizon Self Forcing with DMD distillation, exposing the model to its own rollout distribution while preserving the quality of the pretrained bidirectional teacher. To explicitly control speech progression, an external language model predicts discrete 25-Hz speech-planning tokens, whose continuous features condition the audio diffusion branch and align each causal block with the content it should speak. With bounded causal context and four-step denoising, Vorch-Streamer jointly generates audio and video from text at 27.12 FPS, exceeding the 24-FPS real-time playback rate while maintaining competitive audio-lip synchronization and strong identity preservation over long-form generation.

Related papers

• DreamX-Creator: Democratizing Native Audio-Video Generation at 2K Resolution [16.570672947240855]
  We present DreamX-Creator 1.0, a compact native joint audio-video generation system centered on a 7B generator.<n>A unified Audio-Video Data System constructs and filters temporally coherent clips, produces structured multimodal annotations, and organizes clips into capability-oriented data pools.<n>For high-resolution output, our Autoregressive 1-Step 2K Refinement pipeline adapts a bidirectional multi-step teacher into an autoregressive multi-step refiner.
  arXiv  Detail & Related papers  (2026-08-31T17:11:05Z)
• Don't Pause: Streaming Video-Language Synchrony for Online Video Understanding [69.296913137409]
  We introduce a novel paradigm for online video understanding: Streaming Video-Language Synchrony (SVLS)<n>We present LyraV, a live streaming assistant built upon a hierarchical control framework with two core innovations.<n>First, the Frame-Driven Transition Controller (FDTC), makes high-level semantic decisions on when to continue speaking, start a new response, or stay silent.<n>Second, the Streaming Token Pacer (SToP), a plug-and-play lightweight predictive module, dynamically adapts the language generation rate to match the pace of the visual content.
  arXiv  Detail & Related papers  (2026-06-05T07:29:20Z)
• StreamChar: Long-Horizon Streaming Character Audio-Video Generation with Decoupled Orchestration [16.23723735702324]
  We present StreamChar, a streaming framework that separates long-horizon orchestration from short-window audio-video denoising.<n>Experiments on short-clip and long-horizon protocols show that StreamChar runs in real time on a single H100 GPU.
  arXiv  Detail & Related papers  (2026-05-25T10:04:52Z)
• DiFlowDubber: Discrete Flow Matching for Automated Video Dubbing via Cross-Modal Alignment and Synchronization [16.192222723269925]
  Video dubbing has broad applications in filmmaking, multimedia creation, and assistive speech technology.<n>Existing approaches either train directly on limited dubbing datasets or adopt a two-stage pipeline that adapts pre-trained text-to-speech (TTS) models.<n>We propose DiFlowDubber with a novel two-stage training framework that effectively transfers knowledge from a pre-trained TTS model to video-driven dubbing.<n>Experiments on two primary benchmark datasets demonstrate that DiFlowDubber outperforms previous methods across multiple metrics.
  arXiv  Detail & Related papers  (2026-03-15T07:53:23Z)
• LTX-2: Efficient Joint Audio-Visual Foundation Model [3.1804093402153506]
  LTX-2 is an open-source model capable of generating temporally synchronized audiovisual content.<n>We employ a multilingual text encoder for broader prompt understanding.<n>LTX-2 produces rich, coherent audio tracks that follow the characters, environment, style, and emotion of each scene.
  arXiv  Detail & Related papers  (2026-01-06T18:24:41Z)
• StreamingTalker: Audio-driven 3D Facial Animation with Autoregressive Diffusion Model [73.30619724574642]
  Speech-driven 3D facial animation aims to generate realistic and synchronized facial motions driven by speech inputs.<n>Recent methods have employed audio-conditioned diffusion models for 3D facial animation.<n>We propose a novel autoregressive diffusion model that processes audio in a streaming manner.
  arXiv  Detail & Related papers  (2025-11-18T07:55:16Z)
• StableAvatar: Infinite-Length Audio-Driven Avatar Video Generation [91.45910771331741]
  Current diffusion models for audio-driven avatar video generation struggle to synthesize long videos with natural audio synchronization and identity consistency.<n>This paper presents StableAvatar, the first end-to-end video diffusion transformer that synthesizes infinite-length high-quality videos without post-processing.
  arXiv  Detail & Related papers  (2025-08-11T17:58:24Z)
• READ: Real-time and Efficient Asynchronous Diffusion for Audio-driven Talking Head Generation [55.58089937219475]
  We propose READ, the first real-time diffusion-transformer-based talking head generation framework.<n>Our approach first learns highly compressed video latent space via a VAE, significantly reducing the token count to speech generation.<n>We show that READ outperforms state-of-the-art methods by generating competitive talking head videos with significantly reduced runtime.
  arXiv  Detail & Related papers  (2025-08-05T13:57:03Z)
• Mirasol3B: A Multimodal Autoregressive model for time-aligned and contextual modalities [67.89368528234394]
  One of the main challenges of multimodal learning is the need to combine heterogeneous modalities. Video and audio are obtained at much higher rates than text and are roughly aligned in time. Our approach achieves the state-of-the-art on well established multimodal benchmarks, outperforming much larger models.
  arXiv  Detail & Related papers  (2023-11-09T19:15:12Z)
• Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models [65.18102159618631]
  multimodal generative modeling has created milestones in text-to-image and text-to-video generation. Its application to audio still lags behind for two main reasons: the lack of large-scale datasets with high-quality text-audio pairs, and the complexity of modeling long continuous audio data. We propose Make-An-Audio with a prompt-enhanced diffusion model that addresses these gaps.
  arXiv  Detail & Related papers  (2023-01-30T04:44:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.