OmniForcing: Unleashing Real-time Joint Audio-Visual Generation

• URL: http://arxiv.org/abs/2603.11647v2
• Date: Fri, 13 Mar 2026 04:45:19 GMT
• Title: OmniForcing: Unleashing Real-time Joint Audio-Visual Generation
• Abstract summary: Recent joint audio-visual diffusion models achieve remarkable generation quality but suffer from high latency.<n>We propose OmniForcing, the first framework to distill an offline, dual-stream bidirectional diffusion model into a high-fidelity streaming autore generator.
• Score: 51.031726911410594
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent joint audio-visual diffusion models achieve remarkable generation quality but suffer from high latency due to their bidirectional attention dependencies, hindering real-time applications. We propose OmniForcing, the first framework to distill an offline, dual-stream bidirectional diffusion model into a high-fidelity streaming autoregressive generator. However, naively applying causal distillation to such dual-stream architectures triggers severe training instability, due to the extreme temporal asymmetry between modalities and the resulting token sparsity. We address the inherent information density gap by introducing an Asymmetric Block-Causal Alignment with a zero-truncation Global Prefix that prevents multi-modal synchronization drift. The gradient explosion caused by extreme audio token sparsity during the causal shift is further resolved through an Audio Sink Token mechanism equipped with an Identity RoPE constraint. Finally, a Joint Self-Forcing Distillation paradigm enables the model to dynamically self-correct cumulative cross-modal errors from exposure bias during long rollouts. Empowered by a modality-independent rolling KV-cache inference scheme, OmniForcing achieves state-of-the-art streaming generation at $\sim$25 FPS on a single GPU, maintaining multi-modal synchronization and visual quality on par with the bidirectional teacher.\textbf{Project Page:} \href{https://omniforcing.com}{https://omniforcing.com}

Related papers

• LiveVVT: High-Fidelity Video Virtual Try-On in Real Time [36.4434648528291]
  Video Virtual Try-On (VVT) achieves high visual fidelity through bidirectional modeling.<n>We introduce LiveVVT, a rolling streaming diffusion framework that preserves bidirectional bounded modeling.<n>We show that Experiments on paired and unpaired long-sequence benchmarks demonstrate superior generation quality over similarly sized models.
  arXiv  Detail & Related papers  (2026-08-27T07:06:51Z)
• LeapTalk: Breaking the Latency-Quality Trade-off in Talking Head Generation [28.818825505825476]
  We propose LeapTalk, a novel framework that achieves stable and real-time talking-head generation with a single forward step.<n>Our method achieves high-fidelity and temporally consistent video generation with only 1 step at up to 200 FPS.
  arXiv  Detail & Related papers  (2026-07-29T14:44:57Z)
• Flex-Forcing: Towards a Unified Autoregressive and Bidirectional Video Diffusion Model [88.42070387375679]
  We introduce Flex-Forcing, a unified training and inference framework that enables a video diffusion model to seamlessly operate under both bidirectional and autoregressive generation regimes.<n>Experiments on multiple video generation benchmarks demonstrate that Flex-Forcing achieves consistently better video quality, long-video stability than strong baselines with a rigid inference schedule.
  arXiv  Detail & Related papers  (2026-07-03T17:34:14Z)
• AAD-1: Asymmetric Adversarial Distillation for One-Step Autoregressive Video Generation [66.55343616132005]
  AAD-1 is an Asymmetric Adversarial Distillation framework for one-step autoregressive image-to-video generation.<n>AAD-1 achieves state-of-the-art performance in one-step autoregressive video generation.
  arXiv  Detail & Related papers  (2026-06-02T17:55:30Z)
• AvatarForcing: One-Step Streaming Talking Avatars via Local-Future Sliding-Window Denoising [15.787466786514164]
  AvatarForcing is a one-step streaming diffusion framework that denoises a fixed local-future window with heterogeneous noise levels.<n>Experiments on standard benchmarks and a new 400-video long-form benchmark show strong visual quality and lip synchronization at 34 ms/frame.
  arXiv  Detail & Related papers  (2026-03-15T11:42:07Z)
• From Flow to One Step: Real-Time Multi-Modal Trajectory Policies via Implicit Maximum Likelihood Estimation-based Distribution Distillation [18.70033095161235]
  We propose a framework that distills a Conditional Flow Matching expert into a fast single-step student via Implicit Likelihood Estimation (IMLE)<n>A bi-directional Chamfer distance provides a set-level objective that promotes both mode coverage and fidelity.<n>A unified perception encoder further integrates multi-view RGB, depth, point clouds, and proprioception into a geometry-aware representation.
  arXiv  Detail & Related papers  (2026-03-10T09:30:05Z)
• HiAR: Efficient Autoregressive Long Video Generation via Hierarchical Denoising [52.237486207964245]
  HiAR is a hierarchical denoising framework that reverses the conventional generation order.<n>It performs causal generation across all blocks at every denoising step, so that each block is always conditioned on context at the same noise level.<n>On VBench (20s generation), HiAR achieves the best overall score and the lowest temporal drift among all compared methods.
  arXiv  Detail & Related papers  (2026-03-09T17:58:16Z)
• EchoTorrent: Towards Swift, Sustained, and Streaming Multi-Modal Video Generation [8.795438456031512]
  Multi-modal generation models have achieved high visual quality, but their prohibitive latency and limited temporal stability hinder real-time deployment.<n> Streaming inference exacerbates these issues, leading to pronounced multimodal ambiguities, such as blurring, temporal drift, and lip dechronization.<n>We propose EchoTorrent, a novel novel with a fourfold schema: Multi-Teacher Training fine-tunes a pre-trained model on distinct preference domains; Adaptive DMD (ACCDMD) calibrates the audio CFG degradation errors in phased via a schedule; Long Hybrid Tail, which enforces alignment exclusively on tail frames during long-horizon self-roll
  arXiv  Detail & Related papers  (2026-02-14T08:32:38Z)
• SoulX-FlashTalk: Real-Time Infinite Streaming of Audio-Driven Avatars via Self-Correcting Bidirectional Distillation [16.34443339642213]
  textbfX-FlashTalk is a 14B-scale system to achieve a textbfsub-second start-up latency (0.87s) while reaching a real-time throughput of textbf32 FPS.<n>SoulX-FlashTalk is the first 14B-scale system to achieve a textbfsub-second start-up latency (0.87s) while reaching a real-time throughput of textbf32 FPS.
  arXiv  Detail & Related papers  (2025-12-29T11:18:24Z)
• DiffusionDriveV2: Reinforcement Learning-Constrained Truncated Diffusion Modeling in End-to-End Autonomous Driving [65.7087560656003]
  Generative diffusion models for end-to-end autonomous driving often suffer from mode collapse.<n>We propose DiffusionDriveV2, which leverages reinforcement learning to constrain low-quality modes and explore for superior trajectories.<n>This significantly enhances the overall output quality while preserving the inherent multimodality of its core Gaussian Mixture Model.
  arXiv  Detail & Related papers  (2025-12-08T17:29:52Z)
• Towards One-step Causal Video Generation via Adversarial Self-Distillation [71.30373662465648]
  Recent hybrid video generation models combine autoregressive temporal dynamics with diffusion-based spatial denoising.<n>Our framework produces a single distilled model that flexibly supports multiple inference-step settings.
  arXiv  Detail & Related papers  (2025-11-03T10:12:47Z)
• Rolling Forcing: Autoregressive Long Video Diffusion in Real Time [86.40480237741609]
  Rolling Forcing is a novel video generation technique that enables streaming long videos with minimal error accumulation.<n>Rolling Forcing comes with three novel designs. First, instead of iteratively sampling individual frames, which accelerates error propagation, we design a joint denoising scheme.<n>Second, we introduce the attention sink mechanism into the long-horizon stream video generation task, which allows the model to keep key value states of initial frames as a global context anchor.<n>Third, we design an efficient training algorithm that enables few-step distillation over largely extended denoising windows.
  arXiv  Detail & Related papers  (2025-09-29T17:57:14Z)
• StableDub: Taming Diffusion Prior for Generalized and Efficient Visual Dubbing [63.72095377128904]
  The visual dubbing task aims to generate mouth movements synchronized with the driving audio.<n>Audio-only driving paradigms inadequately capture speaker-specific lip habits.<n>Blind-inpainting approaches produce visual artifacts when handling obstructions.
  arXiv  Detail & Related papers  (2025-09-26T05:23:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.