StreamChar: Long-Horizon Streaming Character Audio-Video Generation with Decoupled Orchestration

• URL: http://arxiv.org/abs/2605.25659v1
• Date: Mon, 25 May 2026 10:04:52 GMT
• Title: StreamChar: Long-Horizon Streaming Character Audio-Video Generation with Decoupled Orchestration
• Abstract summary: We present StreamChar, a streaming framework that separates long-horizon orchestration from short-window audio-video denoising.<n>Experiments on short-clip and long-horizon protocols show that StreamChar runs in real time on a single H100 GPU.
• Score: 16.23723735702324
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Real-time streaming joint audio-video generation for character animation requires a generator to speak the requested transcript, maintain visual identity across chunks, and run within a strict playback budget. These requirements are difficult to satisfy simultaneously: chunk-wise autoregressive generation can accumulate transcript-audio misalignment and visual drift, while the few-step distillation needed for low latency often degrades spatial diversity and temporal quality. We present StreamChar, a streaming framework that separates long-horizon orchestration from short-window audio-video denoising. An LLM-based orchestrator uses the transcript and historical context to produce frame-aligned audio conditions, and a joint audio-video DiT performs local bidirectional denoising with reference and motion-frame conditioning. For efficient deployment, we use a two-stage distillation pipeline that first compresses the sampler and then fine-tunes the student under online chunk rollouts. A progress-aware pointer aligns partial transcripts with generated audio during rollout training, and a sink-chunk memory provides a persistent visual anchor for reducing long-horizon drift. Experiments on short-clip and long-horizon protocols show that StreamChar runs in real time on a single H100 GPU and provides a favorable system-level trade-off among transcript fidelity, audio-visual synchronization, visual quality, and streaming stability compared with recent joint and audio-driven baselines.

Related papers

• Encore: Infinite Audio-Video Generation with Adaptive Signal Routing [22.689981874924943]
  Existing audio-video generation methods produce well-synchronized clips but are limited to short durations.<n>Long-video generation methods extend duration through chunk-based iterative synthesis yet lack audio entirely.<n>We present Encore for long-form synchronized audio-video generation.
  arXiv  Detail & Related papers  (2026-08-28T18:10:18Z)
• Efficient Audio-Visual Generation via Synchrony-Aware Cross-Modal Sparse Attention [31.052048543229464]
  We present a synchronization-aware acceleration framework for efficient audio-visual generation.<n>Our method improves inference efficiency while keeping video quality, audio quality, and audio-video synchronization.
  arXiv  Detail & Related papers  (2026-08-16T04:26:05Z)
• Vorch-Streamer: Extending Human Audio-Visual Generation to Real-Time Long-Form Streaming [24.73791931334844]
  Vorch-Streamer is a post-training framework that enables real-time long-form Text-to-Audio-Video (T2AV) streaming.<n>We construct a synthetic corpus of 80K avatar clips spanning 12-21 seconds and first train a causal generator with mixed Teacher Forcing and Diffusion Forcing.<n>With bounded causal context and four-step denoising, Vorch-Streamer jointly generates audio and video from text at 27.12 FPS, exceeding the 24-FPS real-time playback rate.
  arXiv  Detail & Related papers  (2026-08-06T07:00:37Z)
• Ripple: Real-Time Streaming Audio-Video Generation With Cross-Modal Recurrent Memory [20.92714383117306]
  Ripple is a real-time joint audio-video generation system with a cross-modal recurrent memory mechanism.<n>It achieves 28 FPS at 480P resolution, over faster than the teacher, while capable of coherent long-form generation.
  arXiv  Detail & Related papers  (2026-07-29T12:13:01Z)
• Don't Pause: Streaming Video-Language Synchrony for Online Video Understanding [69.296913137409]
  We introduce a novel paradigm for online video understanding: Streaming Video-Language Synchrony (SVLS)<n>We present LyraV, a live streaming assistant built upon a hierarchical control framework with two core innovations.<n>First, the Frame-Driven Transition Controller (FDTC), makes high-level semantic decisions on when to continue speaking, start a new response, or stay silent.<n>Second, the Streaming Token Pacer (SToP), a plug-and-play lightweight predictive module, dynamically adapts the language generation rate to match the pace of the visual content.
  arXiv  Detail & Related papers  (2026-06-05T07:29:20Z)
• DiscoForcing: A Unified Framework for Real-Time Audio-Driven Character Control with Diffusion Forcing [27.626489251186968]
  We introduce DiscoForcing, a streaming audio-driven diffusion framework.<n>It captures rhythmic structure and phase dynamics with a diffusion-forcing sequence model trained under heterogeneous noise levels.<n>It delivers more stable long-horizon rollouts and sharper audio-motion alignment than prior baselines under matched causality and latency constraints.
  arXiv  Detail & Related papers  (2026-05-27T13:47:40Z)
• Foley-Flow: Coordinated Video-to-Audio Generation with Masked Audio-Visual Alignment and Dynamic Conditional Flows [75.44753202066171]
  Coordinated audio generation based on video inputs typically requires a strict audio-visual (AV) alignment.<n>We propose FoleyFlow to first align unimodal AV encoders via masked modeling training.<n>After training, the AV encoders which are separately pretrained using only unimodal data are aligned with semantic and rhythmic consistency.
  arXiv  Detail & Related papers  (2026-03-09T09:06:25Z)
• Voxtral Realtime [134.66962524291424]
  Voxtral Realtime is a streaming automatic speech recognition model.<n>It matches offline transcription quality at sub-second latency.<n>We release the model weights under the Apache 2.0 license.
  arXiv  Detail & Related papers  (2026-02-11T19:17:10Z)
• StableAvatar: Infinite-Length Audio-Driven Avatar Video Generation [91.45910771331741]
  Current diffusion models for audio-driven avatar video generation struggle to synthesize long videos with natural audio synchronization and identity consistency.<n>This paper presents StableAvatar, the first end-to-end video diffusion transformer that synthesizes infinite-length high-quality videos without post-processing.
  arXiv  Detail & Related papers  (2025-08-11T17:58:24Z)
• StreamMel: Real-Time Zero-shot Text-to-Speech via Interleaved Continuous Autoregressive Modeling [50.537794606598254]
  StreamMel is a pioneering single-stage streaming TTS framework that models continuous mel-spectrograms.<n>It enables low-latency, autoregressive synthesis while preserving high speaker similarity and naturalness.<n>It even achieves performance comparable to offline systems while supporting efficient real-time generation.
  arXiv  Detail & Related papers  (2025-06-14T16:53:39Z)
• Video-Foley: Two-Stage Video-To-Sound Generation via Temporal Event Condition For Foley Sound [19.694770666874827]
  Foley sound synthesis is crucial for multimedia production, enhancing user experience by synchronizing audio and video.<n>Recent studies on automating this labor-intensive process through video-to-sound generation face significant challenges.<n>We propose Video-Foley, a video-to-sound system using Root Mean Square (RMS) as an intuitive condition with semantic timbre prompts.
  arXiv  Detail & Related papers  (2024-08-21T18:06:15Z)
• FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Sounds [14.636030346325578]
  We study Neural Foley, the automatic generation of high-quality sound effects synchronizing with videos, enabling an immersive audio-visual experience. We propose FoleyCrafter, a novel framework that leverages a pre-trained text-to-audio model to ensure high-quality audio generation. One notable advantage of FoleyCrafter is its compatibility with text prompts, enabling the use of text descriptions to achieve controllable and diverse video-to-audio generation according to user intents.
  arXiv  Detail & Related papers  (2024-07-01T17:35:56Z)
• Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow Matching [51.70360630470263]
  Video-to-audio (V2A) generation aims to synthesize content-matching audio from silent video.<n>We propose Frieren, a V2A model based on rectified flow matching.<n>Experiments indicate that Frieren achieves state-of-the-art performance in both generation quality and temporal alignment.
  arXiv  Detail & Related papers  (2024-06-01T06:40:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.