Hallo-Live: Real-Time Streaming Joint Audio-Video Avatar Generation with Asynchronous Dual-Stream and Human-Centric Preference Distillation

• URL: http://arxiv.org/abs/2604.23632v1
• Date: Sun, 26 Apr 2026 09:46:58 GMT
• Title: Hallo-Live: Real-Time Streaming Joint Audio-Video Avatar Generation with Asynchronous Dual-Stream and Human-Centric Preference Distillation
• Abstract summary: We present Hallo-Live, a streaming framework for joint audio-visual avatar generation.<n>Future-Expanding Attention allows each video block to access synchronous audio together with a short horizon of future phonetic cues.<n>HP-DMD reweights training samples using rewards from visual fidelity, speech naturalness, and audio-visual synchronization.
• Score: 29.637920236800394
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Real-time text-driven joint audio-video avatar generation requires jointly synthesizing portrait video and speech with high fidelity and precise synchronization, yet existing audio-visual diffusion models remain too slow for interactive use and often degrade noticeably after aggressive acceleration. We present Hallo-Live, a streaming framework for joint audio-visual avatar generation that combines asynchronous dual-stream diffusion with human-centric preference-guided distillation. To reduce articulation lag in causal generation, we introduce Future-Expanding Attention, which allows each video block to access synchronous audio together with a short horizon of future phonetic cues. To mitigate the quality loss of few-step distillation, we further propose Human-Centric Preference-Guided DMD (HP-DMD), which reweights training samples using rewards from visual fidelity, speech naturalness, and audio-visual synchronization. On two NVIDIA H200 GPUs, Hallo-Live runs at 20.38 FPS with 0.94 seconds latency, yielding 16.0x higher throughput and 99.3x lower latency than the teacher model Ovi. Despite this speedup, it retains strong generation quality, reaching comparable VideoAlign overall score and Sync Confidence score while outperforming other accelerated baselines in the overall quality-efficiency trade-off. Qualitative results further show robust generalization across photorealistic, multi-speaker, and stylized scenarios. To the best of our knowledge, Hallo-Live is the first framework to combine streaming dual-stream diffusion with preference-guided distillation for real-time, text-driven audio-visual generation.

Related papers

• Omni-LiveAvatar: Minute-Level Real-Time Streaming Joint Audio-Visual Avatar Generation [26.625772912688387]
  We present Omni-LiveAvatar, the first framework for minute-level, real-time streaming joint audio-video avatar generation.<n>Experiments show that Omni-LiveAvatar generates high-quality, synchronized minute-level avatars in real time.
  arXiv  Detail & Related papers  (2026-08-07T15:34:15Z)
• Vorch-Streamer: Extending Human Audio-Visual Generation to Real-Time Long-Form Streaming [24.73791931334844]
  Vorch-Streamer is a post-training framework that enables real-time long-form Text-to-Audio-Video (T2AV) streaming.<n>We construct a synthetic corpus of 80K avatar clips spanning 12-21 seconds and first train a causal generator with mixed Teacher Forcing and Diffusion Forcing.<n>With bounded causal context and four-step denoising, Vorch-Streamer jointly generates audio and video from text at 27.12 FPS, exceeding the 24-FPS real-time playback rate.
  arXiv  Detail & Related papers  (2026-08-06T07:00:37Z)
• Speed by Simplicity: A Single-Stream Architecture for Fast Audio-Video Generative Foundation Model [52.79436545460808]
  We present daVinci-MagiHuman, an open-source audio-video generative foundation model for human-centric generation.<n>DaVinci-MagiHuman jointly generates synchronized video and audio using a single-stream Transformer.<n>It supports multilingual spoken generation across Chinese (Mandarin and Cantonese), English, Japanese, Korean, German, and French.
  arXiv  Detail & Related papers  (2026-03-23T13:49:06Z)
• ALIVE: Animate Your World with Lifelike Audio-Video Generation [50.693986608051716]
  ALIVE is a generation model that adapts a pretrained Text-to-Video (T2V) model to Sora-style audio-video generation and animation.<n>To support the audio-visual synchronization and reference animation, we augment the popular MMDiT architecture with a joint audio-video branch.<n>ALIVE demonstrates outstanding performance, consistently outperforming open-source models and matching or surpassing state-of-the-art commercial solutions.
  arXiv  Detail & Related papers  (2026-02-09T14:06:03Z)
• Live Avatar: Streaming Real-time Audio-Driven Avatar Generation with Infinite Length [57.458450695137664]
  We present Live Avatar, an algorithm-system co-designed framework for efficient, high-fidelity, and infinite-length avatar generation.<n>Live Avatar is first to achieve practical, real-time, high-fidelity avatar generation at this scale.
  arXiv  Detail & Related papers  (2025-12-04T11:11:24Z)
• StreamDiffusionV2: A Streaming System for Dynamic and Interactive Video Generation [65.90400162290057]
  Generative models are reshaping the live-streaming industry by redefining how content is created, styled, and delivered.<n>Recent advances in video diffusion have markedly improved temporal consistency and sampling efficiency for offline generation.<n>Live online streaming operates under strict service-level objectives (SLOs): time-to-first-frame must be minimal, and every frame must meet a per-frame deadline with low jitter.
  arXiv  Detail & Related papers  (2025-11-10T18:51:28Z)
• SoundReactor: Frame-level Online Video-to-Audio Generation [39.113214321291586]
  Video-to-Audio generation models operate offline, assuming an entire video sequence or chunks of frames are available beforehand.<n>We introduce the novel task of frame-level online V2A generation, where a model autoregressively generates audio from video without access to future video frames.<n>SoundReactor is the first simple yet effective framework explicitly tailored for this task.
  arXiv  Detail & Related papers  (2025-10-02T15:18:00Z)
• Syncphony: Synchronized Audio-to-Video Generation with Diffusion Transformers [19.226787997122987]
  We present Syncphony, which generates 380x640 resolution, 24fps videos synchronized with diverse audio inputs.<n>Our approach builds upon a pre-trained video backbone and incorporates two key components to improve synchronization.<n>Experiments on AVSync15 and The Greatest Hits datasets demonstrate that Syncphony outperforms existing methods in both synchronization accuracy and visual quality.
  arXiv  Detail & Related papers  (2025-09-26T05:30:06Z)
• StableAvatar: Infinite-Length Audio-Driven Avatar Video Generation [91.45910771331741]
  Current diffusion models for audio-driven avatar video generation struggle to synthesize long videos with natural audio synchronization and identity consistency.<n>This paper presents StableAvatar, the first end-to-end video diffusion transformer that synthesizes infinite-length high-quality videos without post-processing.
  arXiv  Detail & Related papers  (2025-08-11T17:58:24Z)
• READ: Real-time and Efficient Asynchronous Diffusion for Audio-driven Talking Head Generation [55.58089937219475]
  We propose READ, the first real-time diffusion-transformer-based talking head generation framework.<n>Our approach first learns highly compressed video latent space via a VAE, significantly reducing the token count to speech generation.<n>We show that READ outperforms state-of-the-art methods by generating competitive talking head videos with significantly reduced runtime.
  arXiv  Detail & Related papers  (2025-08-05T13:57:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.