DiscoForcing: A Unified Framework for Real-Time Audio-Driven Character Control with Diffusion Forcing

• URL: http://arxiv.org/abs/2605.28491v1
• Date: Wed, 27 May 2026 13:47:40 GMT
• Title: DiscoForcing: A Unified Framework for Real-Time Audio-Driven Character Control with Diffusion Forcing
• Abstract summary: We introduce DiscoForcing, a streaming audio-driven diffusion framework.<n>It captures rhythmic structure and phase dynamics with a diffusion-forcing sequence model trained under heterogeneous noise levels.<n>It delivers more stable long-horizon rollouts and sharper audio-motion alignment than prior baselines under matched causality and latency constraints.
• Score: 27.626489251186968
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study real-time audio-responsive character control as a deployment-faithful problem: strictly causal, bounded-latency streaming that must generate coherent full-body motion at interactive frame rates while the audio condition can change abruptly, including tempo shifts, drops, or user edits. Prior music-to-motion systems are largely optimized for offline generation with global context, and degrade in streaming rollouts where conditioning history becomes stale or unreliable. We introduce DiscoForcing, a streaming audio-driven diffusion framework that combines a causal music encoder that captures rhythmic structure and phase dynamics with a diffusion-forcing sequence model trained under heterogeneous noise levels across the temporal horizon. Building on this, we design a hybrid temporal schedule and a history-guided streaming sampler to explicitly trade off responsiveness against long-horizon consistency under non-stationary audio. Implemented in an end-to-end real-time interactive system with online avatar playback and humanoid deployment workflows, DiscoForcing delivers more stable long-horizon rollouts and sharper audio-motion alignment than prior baselines under matched causality and latency constraints while maintaining real-time throughput.

Related papers

• EchoAvatar: Real-time Generative Avatar Animation from Audio Streams [31.328378976492775]
  We introduce a novel framework designed to generate continuous, coherent full-body motion from streaming speech and music with low latency.<n>By combining this controllability with stream audio-driven synthesis, our framework serves as a plug-and-play solution for transforming voice agents into interactive humanoid avatars.
  arXiv  Detail & Related papers  (2026-05-27T10:18:16Z)
• StreamChar: Long-Horizon Streaming Character Audio-Video Generation with Decoupled Orchestration [16.23723735702324]
  We present StreamChar, a streaming framework that separates long-horizon orchestration from short-window audio-video denoising.<n>Experiments on short-clip and long-horizon protocols show that StreamChar runs in real time on a single H100 GPU.
  arXiv  Detail & Related papers  (2026-05-25T10:04:52Z)
• Personalizing Causal Audio-Driven Facial Motion via Dynamic Multi-modal Retrieval [16.85147888618743]
  We present an end-to-end causal framework for personalizing causal facial motion generation.<n>Our method significantly outperforms state-of-the-art approaches in lip-sync accuracy, identity consistency, and perceived realism.
  arXiv  Detail & Related papers  (2026-04-26T13:15:08Z)
• CoSyncDiT: Cognitive Synchronous Diffusion Transformer for Movie Dubbing [76.74048814837336]
  Movie dubbing aims to synthesize speech that preserves the vocal identity of a reference audio while synchronizing with the lip movements in a target video.<n>Existing methods fail to achieve precise lip-sync and lack naturalness due to explicit alignment at the duration level.<n>We propose a novel flow matching-based movie dubbing framework driven by the Cognitive Synchronous Diffusion Transformer (CoSync-DiT)
  arXiv  Detail & Related papers  (2026-04-14T05:03:57Z)
• STRIDE: When to Speak Meets Sequence Denoising for Streaming Video Understanding [77.20037111885226]
  Real-world deployments increasingly require streaming perception and proactive interaction.<n>In this work, we revisit proactive activation in streaming video as a structured sequence modeling problem.<n>We propose STRIDE, which employs a lightweight masked diffusion module at the activation interface to jointly predict and progressively refine activation signals.
  arXiv  Detail & Related papers  (2026-03-29T09:23:45Z)
• OmniForcing: Unleashing Real-time Joint Audio-Visual Generation [51.031726911410594]
  Recent joint audio-visual diffusion models achieve remarkable generation quality but suffer from high latency.<n>We propose OmniForcing, the first framework to distill an offline, dual-stream bidirectional diffusion model into a high-fidelity streaming autore generator.
  arXiv  Detail & Related papers  (2026-03-12T08:17:36Z)
• Knot Forcing: Taming Autoregressive Video Diffusion Models for Real-time Infinite Interactive Portrait Animation [16.692450893925148]
  We present a novel streaming framework named Knot Forcing for real-time portrait animation.<n>K Knot Forcing enables high-fidelity, temporally consistent, and interactive portrait animation over infinite sequences.
  arXiv  Detail & Related papers  (2025-12-25T16:34:56Z)
• LILAC: Long-sequence Incremental Low-latency Arbitrary Motion Stylization via Streaming VAE-Diffusion with Causal Decoding [5.946860384629338]
  LILAC builds upon a recent high-performing offline framework for arbitrary motion stylization.<n>It extends it to an online setting through a latent-space streaming architecture with a sliding-window causal design.<n>This architecture enables real-time arbitrary stylization without relying on future frames or modifying the diffusion model architecture.
  arXiv  Detail & Related papers  (2025-10-17T07:45:43Z)
• InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing [66.48064661467781]
  We introduce sparse-frame video dubbing, a novel paradigm that strategically preserves references to maintain identity, iconic gestures, and camera trajectories.<n>We propose InfiniteTalk, a streaming audio-driven generator designed for infinite-length long sequence dubbing.<n> Comprehensive evaluations on HDTF, CelebV-HQ, and EMTD datasets demonstrate state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-08-19T17:55:23Z)
• StableAvatar: Infinite-Length Audio-Driven Avatar Video Generation [91.45910771331741]
  Current diffusion models for audio-driven avatar video generation struggle to synthesize long videos with natural audio synchronization and identity consistency.<n>This paper presents StableAvatar, the first end-to-end video diffusion transformer that synthesizes infinite-length high-quality videos without post-processing.
  arXiv  Detail & Related papers  (2025-08-11T17:58:24Z)
• READ: Real-time and Efficient Asynchronous Diffusion for Audio-driven Talking Head Generation [55.58089937219475]
  We propose READ, the first real-time diffusion-transformer-based talking head generation framework.<n>Our approach first learns highly compressed video latent space via a VAE, significantly reducing the token count to speech generation.<n>We show that READ outperforms state-of-the-art methods by generating competitive talking head videos with significantly reduced runtime.
  arXiv  Detail & Related papers  (2025-08-05T13:57:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.