Encore: Infinite Audio-Video Generation with Adaptive Signal Routing

• URL: http://arxiv.org/abs/2609.04249v1
• Date: Fri, 28 Aug 2026 18:10:18 GMT
• Title: Encore: Infinite Audio-Video Generation with Adaptive Signal Routing
• Abstract summary: Existing audio-video generation methods produce well-synchronized clips but are limited to short durations.<n>Long-video generation methods extend duration through chunk-based iterative synthesis yet lack audio entirely.<n>We present Encore for long-form synchronized audio-video generation.
• Score: 22.689981874924943
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing audio-video generation methods produce well-synchronized clips but are limited to short durations, while long-video generation methods extend duration through chunk-based iterative synthesis yet lack audio entirely. Generating long audio-video jointly is fundamentally harder than either task alone: each chunk must simultaneously maintain video temporal coherence, audio temporal coherence, and cross-modal synchronization, whose conditioning signals enter the model through different pathways. In this work, we present Encore for long-form synchronized audio-video generation. Our key insight is to factor this challenge into: (1) local continuity handled by iterative generation with explicit cross-chunk context propagation, and (2) global consistency enforced via reference audio-video signals with shifted position embedding. Building on this design, we propose Adaptive Signal Routing (ASR), which introduces learnable attention biases within self-attention and learnable residual scales on cross-attention outputs, enabling the model to adaptively modulate the influence of each conditioning signal. Trained end-to-end for joint audio-video generation, Encore also supports infinite-length audio-to-video and video-to-audio synthesis at inference by conditioning on the ground-truth modality throughout the denoising process. Experiments on our extended VerseBench for long audio-video evaluation demonstrate that Encore significantly outperforms existing methods in both generation quality and temporal coherence. Code and data for this paper are at https://github.com/shaohua-pan/Encore.

Related papers

• Efficient Audio-Visual Generation via Synchrony-Aware Cross-Modal Sparse Attention [31.052048543229464]
  We present a synchronization-aware acceleration framework for efficient audio-visual generation.<n>Our method improves inference efficiency while keeping video quality, audio quality, and audio-video synchronization.
  arXiv  Detail & Related papers  (2026-08-16T04:26:05Z)
• OmniVAE: An Audio-Video VAE with Cross-Modal Alignment for Joint Generation [73.57253966578186]
  We present OmniVAE, a jointly trained audio-video VAE that learns fine-grained semantic alignment between audio and video latent representations.<n>Experiments show that both objectives consistently improve the learnability of the latent spaces.
  arXiv  Detail & Related papers  (2026-07-26T21:51:28Z)
• Diffusion Models for Joint Audio-Video Generation [51.56484100374058]
  I release two high-quality, paired audio-video datasets.<n>I train the MM-Diffusion architecture from scratch on our datasets.<n>I propose a sequential two-step text-to-audio-video generation pipeline.
  arXiv  Detail & Related papers  (2026-03-17T03:31:37Z)
• Foley-Flow: Coordinated Video-to-Audio Generation with Masked Audio-Visual Alignment and Dynamic Conditional Flows [75.44753202066171]
  Coordinated audio generation based on video inputs typically requires a strict audio-visual (AV) alignment.<n>We propose FoleyFlow to first align unimodal AV encoders via masked modeling training.<n>After training, the AV encoders which are separately pretrained using only unimodal data are aligned with semantic and rhythmic consistency.
  arXiv  Detail & Related papers  (2026-03-09T09:06:25Z)
• ViSAudio: End-to-End Video-Driven Binaural Spatial Audio Generation [55.76423101183408]
  ViSAudio is an end-to-end framework that employs conditional flow matching with a dual-branch audio generation architecture.<n>It generates high-quality audio with spatial immersion that adapts to viewpoint changes, sound-source motion, and diverse acoustic environments.
  arXiv  Detail & Related papers  (2025-12-02T18:56:12Z)
• SALSA-V: Shortcut-Augmented Long-form Synchronized Audio from Videos [38.40457780873775]
  We propose SALSA-V, a multimodal video-to-audio generation model capable of synthesizing highly synchronized, high-fidelity long-form audio from silent video content.<n>Our approach introduces a masked diffusion objective, enabling audio-conditioned generation and the seamless synthesis of audio sequences of unconstrained length.<n>We demonstrate that SALSA-V significantly outperforms existing state-of-the-art methods in both audiovisual alignment and synchronization with video content in quantitative evaluation and a human listening study.
  arXiv  Detail & Related papers  (2025-10-03T11:37:50Z)
• StableAvatar: Infinite-Length Audio-Driven Avatar Video Generation [91.45910771331741]
  Current diffusion models for audio-driven avatar video generation struggle to synthesize long videos with natural audio synchronization and identity consistency.<n>This paper presents StableAvatar, the first end-to-end video diffusion transformer that synthesizes infinite-length high-quality videos without post-processing.
  arXiv  Detail & Related papers  (2025-08-11T17:58:24Z)
• End to End Lip Synchronization with a Temporal AutoEncoder [95.94432031144716]
  We study the problem of syncing the lip movement in a video with the audio stream. Our solution finds an optimal alignment using a dual-domain recurrent neural network. As an application, we demonstrate our ability to robustly align text-to-speech generated audio with an existing video stream.
  arXiv  Detail & Related papers  (2022-03-30T12:00:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.