StableAvatar: Infinite-Length Audio-Driven Avatar Video Generation

• URL: http://arxiv.org/abs/2508.08248v1
• Date: Mon, 11 Aug 2025 17:58:24 GMT
• Title: StableAvatar: Infinite-Length Audio-Driven Avatar Video Generation
• Abstract summary: Current diffusion models for audio-driven avatar video generation struggle to synthesize long videos with natural audio synchronization and identity consistency.<n>This paper presents StableAvatar, the first end-to-end video diffusion transformer that synthesizes infinite-length high-quality videos without post-processing.
• Score: 91.45910771331741
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Current diffusion models for audio-driven avatar video generation struggle to synthesize long videos with natural audio synchronization and identity consistency. This paper presents StableAvatar, the first end-to-end video diffusion transformer that synthesizes infinite-length high-quality videos without post-processing. Conditioned on a reference image and audio, StableAvatar integrates tailored training and inference modules to enable infinite-length video generation. We observe that the main reason preventing existing models from generating long videos lies in their audio modeling. They typically rely on third-party off-the-shelf extractors to obtain audio embeddings, which are then directly injected into the diffusion model via cross-attention. Since current diffusion backbones lack any audio-related priors, this approach causes severe latent distribution error accumulation across video clips, leading the latent distribution of subsequent segments to drift away from the optimal distribution gradually. To address this, StableAvatar introduces a novel Time-step-aware Audio Adapter that prevents error accumulation via time-step-aware modulation. During inference, we propose a novel Audio Native Guidance Mechanism to further enhance the audio synchronization by leveraging the diffusion's own evolving joint audio-latent prediction as a dynamic guidance signal. To enhance the smoothness of the infinite-length videos, we introduce a Dynamic Weighted Sliding-window Strategy that fuses latent over time. Experiments on benchmarks show the effectiveness of StableAvatar both qualitatively and quantitatively.

Related papers

• Encore: Infinite Audio-Video Generation with Adaptive Signal Routing [22.689981874924943]
  Existing audio-video generation methods produce well-synchronized clips but are limited to short durations.<n>Long-video generation methods extend duration through chunk-based iterative synthesis yet lack audio entirely.<n>We present Encore for long-form synchronized audio-video generation.
  arXiv  Detail & Related papers  (2026-08-28T18:10:18Z)
• Vorch-Streamer: Extending Human Audio-Visual Generation to Real-Time Long-Form Streaming [24.73791931334844]
  Vorch-Streamer is a post-training framework that enables real-time long-form Text-to-Audio-Video (T2AV) streaming.<n>We construct a synthetic corpus of 80K avatar clips spanning 12-21 seconds and first train a causal generator with mixed Teacher Forcing and Diffusion Forcing.<n>With bounded causal context and four-step denoising, Vorch-Streamer jointly generates audio and video from text at 27.12 FPS, exceeding the 24-FPS real-time playback rate.
  arXiv  Detail & Related papers  (2026-08-06T07:00:37Z)
• JoyAvatar: Real-time and Infinite Audio-Driven Avatar Generation with Autoregressive Diffusion [19.420963062956222]
  JoyAvatar is an audio-driven autoregressive model capable of real-time inference and infinite-length video generation.<n>Our model achieves competitive results in visual quality, temporal consistency, and lip synchronization.
  arXiv  Detail & Related papers  (2025-12-12T10:06:01Z)
• Live Avatar: Streaming Real-time Audio-Driven Avatar Generation with Infinite Length [57.458450695137664]
  We present Live Avatar, an algorithm-system co-designed framework for efficient, high-fidelity, and infinite-length avatar generation.<n>Live Avatar is first to achieve practical, real-time, high-fidelity avatar generation at this scale.
  arXiv  Detail & Related papers  (2025-12-04T11:11:24Z)
• HunyuanVideo-Foley: Multimodal Diffusion with Representation Alignment for High-Fidelity Foley Audio Generation [14.921126281071544]
  HunyuanVideo-Foley is an end-to-end text-video-to-audio framework.<n>It synthesizes high-fidelity audio precisely aligned with visual dynamics and semantic context.<n>It achieves new state-of-the-art performance across audio fidelity, visual-semantic alignment, temporal alignment and distribution matching.
  arXiv  Detail & Related papers  (2025-08-23T07:30:18Z)
• READ: Real-time and Efficient Asynchronous Diffusion for Audio-driven Talking Head Generation [55.58089937219475]
  We propose READ, the first real-time diffusion-transformer-based talking head generation framework.<n>Our approach first learns highly compressed video latent space via a VAE, significantly reducing the token count to speech generation.<n>We show that READ outperforms state-of-the-art methods by generating competitive talking head videos with significantly reduced runtime.
  arXiv  Detail & Related papers  (2025-08-05T13:57:03Z)
• FreeLong: Training-Free Long Video Generation with SpectralBlend Temporal Attention [57.651429116402554]
  This paper investigates a straightforward and training-free approach to extend an existing short video diffusion model for consistent long video generation. We find that directly applying the short video diffusion model to generate long videos can lead to severe video quality degradation. Motivated by this, we propose a novel solution named FreeLong to balance the frequency distribution of long video features during the denoising process.
  arXiv  Detail & Related papers  (2024-07-29T11:52:07Z)
• Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners [69.70590867769408]
  Video and audio content creation serves as the core technique for the movie industry and professional users. Existing diffusion-based methods tackle video and audio generation separately, which hinders the technique transfer from academia to industry. In this work, we aim at filling the gap, with a carefully designed optimization-based framework for cross-visual-audio and joint-visual-audio generation.
  arXiv  Detail & Related papers  (2024-02-27T17:57:04Z)
• FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling [85.60543452539076]
  Existing video generation models are typically trained on a limited number of frames, resulting in the inability to generate high-fidelity long videos during inference. This study explores the potential of extending the text-driven capability to generate longer videos conditioned on multiple texts. We propose FreeNoise, a tuning-free and time-efficient paradigm to enhance the generative capabilities of pretrained video diffusion models.
  arXiv  Detail & Related papers  (2023-10-23T17:59:58Z)
• APLA: Additional Perturbation for Latent Noise with Adversarial Training Enables Consistency [9.07931905323022]
  We propose a novel text-to-video (T2V) generation network structure based on diffusion models. Our approach only necessitates a single video as input and builds upon pre-trained stable diffusion networks. We leverage a hybrid architecture of transformers and convolutions to compensate for temporal intricacies, enhancing consistency between different frames within the video.
  arXiv  Detail & Related papers  (2023-08-24T07:11:00Z)
• Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generation [41.292644854306594]
  We propose a novel diffusion-based framework, named Diffusion Co-Speech Gesture (DiffGesture) DiffGesture achieves state-of-theart performance, which renders coherent gestures with better mode coverage and stronger audio correlations.
  arXiv  Detail & Related papers  (2023-03-16T07:32:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.