EchoAvatar: Real-time Generative Avatar Animation from Audio Streams

• URL: http://arxiv.org/abs/2605.28272v1
• Date: Wed, 27 May 2026 10:18:16 GMT
• Title: EchoAvatar: Real-time Generative Avatar Animation from Audio Streams
• Abstract summary: We introduce a novel framework designed to generate continuous, coherent full-body motion from streaming speech and music with low latency.<n>By combining this controllability with stream audio-driven synthesis, our framework serves as a plug-and-play solution for transforming voice agents into interactive humanoid avatars.
• Score: 31.328378976492775
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Real-time synthesis of high-fidelity 3D character motion from audio is a pivotal component for next-generation interactive avatars and virtual assistants. However, most existing approaches are limited to offline processing of complete audio sequences or are constrained to specific domains, rarely handling both speech and music effectively. In this paper, we introduce a novel framework designed to generate continuous, coherent full-body motion from streaming speech and music with low latency. Central to our approach is a unified streaming architecture capable of synthesizing continuous motion from incremental audio inputs. We employ a robust training strategy that enforces strong audio dependency, allowing the model to seamlessly generalize across conversational speech and rhythmic music without requiring explicit domain labels or mode switching. Additionally, we explored Reinforcement Learning to refine the quality of online generation. Furthermore, we bridge reactive animation with intent-driven behavior via a tool-call interface that allows upstream Large Language Models to inject explicit semantic control. By combining this controllability with stream audio-driven synthesis, our framework serves as a plug-and-play solution for transforming voice agents into interactive humanoid avatars. Extensive experiments demonstrate that our method outperforms state-of-the-art realtime baselines in motion quality and synchronization while maintaining the flexibility required for live deployment. Our code, pre-trained models, and videos are available at https://robinwitch.github.io/EchoAvatar-Page.

Related papers

• OmniMate: Open-Ended Real-Time Streaming Audio-Visual Generation for Interactive Avatars [13.53343552751081]
  We propose a unified framework for open-ended real-time interactive audio-visual avatar generation.<n> OmniMate jointly synthesizes visual content, speech, and sound effects in real time, enabling natural and immersive multi-turn interactions.<n>We show that OmniMate achieves high-quality, low-latency streaming generation while maintaining strong long-term audio-visual consistency.
  arXiv  Detail & Related papers  (2026-07-25T03:42:52Z)
• Semantic Audio-driven Understanding for Dynamic Humanoid Whole Body Control [0.0]
  We introduce a novel multi-modal orchestration framework for semantic audio-driven humanoid control.<n>The system processes continuous audio streams and routes them into music or speech branches.<n>Music input is handled via audio fingerprinting and semantic embeddings to retrieve track identity and temporal alignment.<n>Speech input is grounded into a discrete library of imitation-learned skills, enabling direct human-robot interaction.
  arXiv  Detail & Related papers  (2026-07-15T14:24:00Z)
• Audio Interaction Model [102.4354125819644]
  Today's Large Audio Language Models (LALMs) are offline, and streaming audio models each handle only a single task such as streaming ASR or voice chatting.<n>It is time to unify them into one online LALM: a model that, through an always-on perceive-decide-respond loop, listens to sound, environment, and instructions in real time and reacts on the fly.<n>We formalize this regime as the Audio Interaction Model, and realize it with Audio-Interaction, a unified streaming model that retains offline task execution while adding online general audio instruction following.
  arXiv  Detail & Related papers  (2026-06-03T17:26:11Z)
• Beyond Monologue: Interactive Talking-Listening Avatar Generation with Conversational Audio Context-Aware Kernels [26.18760496276335]
  We develop an interactive virtual agent capable of simultaneously processing dual-stream audio inputs for both talking and listening.<n>Our approach successfully fuses strong temporal alignment with contextual semantics, setting a new state-of-the-art for generating highly natural and responsive interactive digital humans.
  arXiv  Detail & Related papers  (2026-04-11T22:34:21Z)
• StreamingTalker: Audio-driven 3D Facial Animation with Autoregressive Diffusion Model [73.30619724574642]
  Speech-driven 3D facial animation aims to generate realistic and synchronized facial motions driven by speech inputs.<n>Recent methods have employed audio-conditioned diffusion models for 3D facial animation.<n>We propose a novel autoregressive diffusion model that processes audio in a streaming manner.
  arXiv  Detail & Related papers  (2025-11-18T07:55:16Z)
• Audio Driven Real-Time Facial Animation for Social Telepresence [65.66220599734338]
  We present an audio-driven real-time system for animating photorealistic 3D facial avatars with minimal latency.<n>Central to our approach is an encoder model that transforms audio signals into latent facial expression sequences in real time.<n>We capture the rich spectrum of facial expressions necessary for natural communication while achieving real-time performance.
  arXiv  Detail & Related papers  (2025-10-01T17:57:05Z)
• AvatarSync: Rethinking Talking-Head Animation through Phoneme-Guided Autoregressive Perspective [15.69417162113696]
  AvatarSync is an autoregressive framework on phoneme representations that generates realistic talking-head animations from a single reference image.<n>We show that AvatarSync outperforms existing talking-head animation methods in visual fidelity, temporal consistency, and computational efficiency.
  arXiv  Detail & Related papers  (2025-09-15T15:34:02Z)
• InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing [66.48064661467781]
  We introduce sparse-frame video dubbing, a novel paradigm that strategically preserves references to maintain identity, iconic gestures, and camera trajectories.<n>We propose InfiniteTalk, a streaming audio-driven generator designed for infinite-length long sequence dubbing.<n> Comprehensive evaluations on HDTF, CelebV-HQ, and EMTD datasets demonstrate state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-08-19T17:55:23Z)
• ThinkSound: Chain-of-Thought Reasoning in Multimodal Large Language Models for Audio Generation and Editing [47.14083940177122]
  ThinkSound is a novel framework that enables stepwise, interactive audio generation and editing for videos.<n>Our approach decomposes the process into three complementary stages: semantically coherent, interactive object-centric refinement, and targeted editing.<n> Experiments demonstrate that ThinkSound achieves state-of-the-art performance in video-to-audio generation across both audio metrics and CoT metrics.
  arXiv  Detail & Related papers  (2025-06-26T16:32:06Z)
• OmniAvatar: Efficient Audio-Driven Avatar Video Generation with Adaptive Body Animation [11.71823020976487]
  We introduce OmniAvatar, an audio-driven full-body video generation model.<n>It enhances human animation with improved lip-sync accuracy and natural movements.<n>Experiments show it surpasses existing models in both facial and semi-body video generation.
  arXiv  Detail & Related papers  (2025-06-23T17:33:03Z)
• Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners [69.70590867769408]
  Video and audio content creation serves as the core technique for the movie industry and professional users. Existing diffusion-based methods tackle video and audio generation separately, which hinders the technique transfer from academia to industry. In this work, we aim at filling the gap, with a carefully designed optimization-based framework for cross-visual-audio and joint-visual-audio generation.
  arXiv  Detail & Related papers  (2024-02-27T17:57:04Z)
• Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motion [89.01668641930206]
  We present a framework for modeling interactional communication in dyadic conversations. We autoregressively output multiple possibilities of corresponding listener motion. Our method organically captures the multimodal and non-deterministic nature of nonverbal dyadic interactions.
  arXiv  Detail & Related papers  (2022-04-18T17:58:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.