Mutual Forcing: Dual-Mode Self-Evolution for Fast Autoregressive Audio-Video Character Generation

• URL: http://arxiv.org/abs/2604.25819v1
• Date: Tue, 28 Apr 2026 16:28:05 GMT
• Title: Mutual Forcing: Dual-Mode Self-Evolution for Fast Autoregressive Audio-Video Character Generation
• Abstract summary: Mutual Forcing is a framework for fast autoregressive audio-video generation with long-horizon audio-video synchronization.<n>Our approach addresses two key challenges: joint audio-video modeling and fast autoregressive generation.
• Score: 86.69465782855086
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: In this work, we propose Mutual Forcing, a framework for fast autoregressive audio-video generation with long-horizon audio-video synchronization. Our approach addresses two key challenges: joint audio-video modeling and fast autoregressive generation. To ease joint audio-video optimization, we adopt a two-stage training strategy: we first train uni-modal generators and then couple them into a unified audio-video model for joint training on paired data. For streaming generation, we ask whether a native fast causal audio-video model can be trained directly, instead of following existing streaming distillation pipelines that typically train a bidirectional model first and then convert it into a causal generator through multiple distillation stages. Our answer is Mutual Forcing, which builds directly on native autoregressive model and integrates few-step and multi-step generation within a single weight-shared model, enabling self-distillation and improved training-inference consistency. The multi-step mode improves the few-step mode via self-distillation, while the few-step mode generates historical context during training to improve training-inference consistency; because the two modes share parameters, these two effects reinforce each other within a single model. Compared with prior approaches such as Self-Forcing, Mutual Forcing removes the need for an additional bidirectional teacher model, supports more flexible training sequence lengths, reduces training overhead, and allows the model to improve directly from real paired data rather than a fixed teacher. Experiments show that Mutual Forcing matches or surpasses strong baselines that require around 50 sampling steps while using only 4 to 8 steps, demonstrating substantial advantages in both efficiency and quality. The project page is available at https://mutualforcing.github.io.

Related papers

• ROMS-IMLE: A Minimalist Approach to Competitive Single-Step Generative Modelling [6.903463801726018]
  We take a minimalist approach to designing a competitive generative model.<n>For the training objective, we choose Implicit Maximum Likelihood Estimation (IMLE)<n>For the model, we eschew transformers and instead choose a moderately sized convolutional network.
  arXiv  Detail & Related papers  (2026-07-21T17:51:38Z)
• Talker-T2AV: Joint Talking Audio-Video Generation with Autoregressive Diffusion Modeling [59.97930201616015]
  We propose Talker-T2AV, an autoregressive diffusion framework where high-level cross-modal modeling occurs in a shared backbone.<n>A shared autoregressive language model jointly reasons over audio and video in a unified patch-level token space.<n> Experiments on talking portrait benchmarks show Talker-T2AV outperforms dual-branch baselines in lip-sync accuracy, video quality, and audio quality.
  arXiv  Detail & Related papers  (2026-04-26T07:48:47Z)
• TalkingMachines: Real-Time Audio-Driven FaceTime-Style Video via Autoregressive Diffusion Models [2.176487921193175]
  TalkingMachines is an efficient framework that transforms pretrained video generation models into real-time, audio-driven character animators.<n>TalkingMachines enables natural conversational experiences by integrating an audio large language model (LLM) with our video generation foundation model.
  arXiv  Detail & Related papers  (2025-06-03T17:29:28Z)
• From Slow Bidirectional to Fast Autoregressive Video Diffusion Models [48.35054927704544]
  Current video diffusion models achieve impressive generation quality but struggle in interactive applications due to bidirectional attention dependencies.<n>We address this limitation by adapting a pretrained bidirectional diffusion transformer to an autoregressive transformer that generates frames on-the-fly.<n>Our model achieves a total score of 84.27 on the VBench-Long benchmark, surpassing all previous video generation models.
  arXiv  Detail & Related papers  (2024-12-10T18:59:50Z)
• Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow Matching [51.70360630470263]
  Video-to-audio (V2A) generation aims to synthesize content-matching audio from silent video.<n>We propose Frieren, a V2A model based on rectified flow matching.<n>Experiments indicate that Frieren achieves state-of-the-art performance in both generation quality and temporal alignment.
  arXiv  Detail & Related papers  (2024-06-01T06:40:22Z)
• Audio-Driven Dubbing for User Generated Contents via Style-Aware Semi-Parametric Synthesis [123.11530365315677]
  Existing automated dubbing methods are usually designed for Professionally Generated Content (PGC) production. In this paper, we investigate an audio-driven dubbing method that is more feasible for User Generated Content (UGC) production.
  arXiv  Detail & Related papers  (2023-08-31T15:41:40Z)
• MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation [70.74377373885645]
  We propose the first joint audio-video generation framework that brings engaging watching and listening experiences simultaneously. MM-Diffusion consists of a sequential multi-modal U-Net for a joint denoising process by design. Experiments show superior results in unconditional audio-video generation, and zero-shot conditional tasks.
  arXiv  Detail & Related papers  (2022-12-19T14:11:52Z)
• Differentiable Duration Modeling for End-to-End Text-to-Speech [6.571447892202893]
  parallel text-to-speech (TTS) models have recently enabled fast and highly-natural speech synthesis. We propose a differentiable duration method for learning monotonic sequences between input and output. Our model learns to perform high-fidelity synthesis through a combination of adversarial training and matching the total ground-truth duration.
  arXiv  Detail & Related papers  (2022-03-21T15:14:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.