OmniMate: Open-Ended Real-Time Streaming Audio-Visual Generation for Interactive Avatars

• URL: http://arxiv.org/abs/2607.23023v2
• Date: Tue, 28 Jul 2026 02:01:27 GMT
• Title: OmniMate: Open-Ended Real-Time Streaming Audio-Visual Generation for Interactive Avatars
• Abstract summary: We propose a unified framework for open-ended real-time interactive audio-visual avatar generation.<n> OmniMate jointly synthesizes visual content, speech, and sound effects in real time, enabling natural and immersive multi-turn interactions.<n>We show that OmniMate achieves high-quality, low-latency streaming generation while maintaining strong long-term audio-visual consistency.
• Score: 13.53343552751081
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in diffusion-based generative models have enabled real-time audio-driven avatar generation and unified audio-visual synthesis, providing a promising foundation for interactive avatar systems. However, extending unified audio-visual synthesis to real-time interactive streaming remains challenging, as the generation horizon is unknown in advance and the generated identity may drift over long-term generation. To address these challenges, we propose OmniMate, a unified framework for open-ended real-time interactive audio-visual avatar generation. OmniMate jointly synthesizes visual content, speech, and sound effects in real time, enabling natural and immersive multi-turn interactions. To achieve adaptive response progression, we introduce a Generation Progress Controller (GPC) that explicitly models the generation progress of each streaming chunk, allowing the model to complete responses according to the desired progress and achieve seamless transitions between execution and listening states. To preserve long-term cross-modal identity consistency, we propose a Multi-Reference Conditioning Module (MRCM), which leverages multiple reference images and a reference speech segment to provide persistent visual and speaker identity cues throughout long-duration streaming interactions. Extensive experiments on an interaction-oriented adaptation of VerseBench demonstrate that OmniMate achieves high-quality, low-latency streaming generation while maintaining strong long-term audio-visual consistency. The results further show that OmniMate supports realistic, coherent, and responsive interactive avatar experiences over extended multi-turn conversations.

Related papers

• Long-Horizon Audio-Visual Generation for Persistent Stories and Interactive Worlds [60.18437892555415]
  We present JoyAI-Echo-1.5, a unified audio-visual generation system with two purpose-built variants.<n>The long-video variant introduces composable cross-shot memory that aggregates visual evidence across multiple prior shots and speaker cues derived from speech-filtered full-shot audio.<n>The world-model variant converts heterogeneous navigation inputs into calibrated metric 6-DoF camera trajectories and injects them through a geometry-aware conditioning pathway.
  arXiv  Detail & Related papers  (2026-08-24T15:31:12Z)
• InteractiveAvatar: Real-Time Streaming Video Generation for Consistent and Intent-Aware Avatars [39.5461462800725]
  We propose a real-time infinite-streaming video generation framework that supports visually consistent avatar video generation and intent-aware interactions.<n>With autoregressive distillation, InteractiveAvatar achieves real-time str-eaming generation of human avatars over arbitrarily long durations.<n>Our method achieves state-of-the-art visual consistency in long-duration generation, while enabling complex user-avatar interaction in real time.
  arXiv  Detail & Related papers  (2026-06-22T06:41:17Z)
• EchoAvatar: Real-time Generative Avatar Animation from Audio Streams [31.328378976492775]
  We introduce a novel framework designed to generate continuous, coherent full-body motion from streaming speech and music with low latency.<n>By combining this controllability with stream audio-driven synthesis, our framework serves as a plug-and-play solution for transforming voice agents into interactive humanoid avatars.
  arXiv  Detail & Related papers  (2026-05-27T10:18:16Z)
• Omni-Customizer: End-to-End MultiModal Customization for Joint Audio-Video Generation [93.44732526074876]
  We present an end-to-end framework targeted at the precise binding and seamless fusion of multiple identities.<n>Within this architecture, we propose Semantic-Anchored Multimodal Ro (SA-MRo) to anchor visual and audio reference tokens, along with TTS embeddings, to their corresponding semantic descriptions.<n>Experiments demonstrate that Omni-Context achieves state-of-the-art performance in dual-modal customized generation.
  arXiv  Detail & Related papers  (2026-05-17T14:56:52Z)
• Beyond Monologue: Interactive Talking-Listening Avatar Generation with Conversational Audio Context-Aware Kernels [26.18760496276335]
  We develop an interactive virtual agent capable of simultaneously processing dual-stream audio inputs for both talking and listening.<n>Our approach successfully fuses strong temporal alignment with contextual semantics, setting a new state-of-the-art for generating highly natural and responsive interactive digital humans.
  arXiv  Detail & Related papers  (2026-04-11T22:34:21Z)
• U-Mind: A Unified Framework for Real-Time Multimodal Interaction with Audiovisual Generation [48.6868174403074]
  We introduce U-Mind, the first unified system for high-intelligence multimodal dialogue.<n>It supports real-time generation and jointly models language, speech, motion, and video synthesis within a single interactive loop.<n>We show that U-Mind achieves state-of-the-art performance on a range of multimodal interaction tasks.
  arXiv  Detail & Related papers  (2026-02-27T07:07:02Z)
• MAViD: A Multimodal Framework for Audio-Visual Dialogue Understanding and Generation [59.23161833385837]
  We propose MAViD, a novel Multimodal framework for Audio-Visual Dialogue understanding and generation.<n>Our framework can generate vivid and contextually coherent long-duration dialogue interactions and accurately interpret users' multimodal queries.
  arXiv  Detail & Related papers  (2025-12-02T18:55:53Z)
• X-Streamer: Unified Human World Modeling with Audiovisual Interaction [36.50697656708077]
  X-Streamer is a framework for building digital human agents capable of infinite interactions across text, speech, and video.<n>At its core is a Thinker-Actor dual-transformer architecture that unifies multimodal understanding and generation.<n>X-Streamer runs in real time on two A100 GPUs, sustaining hours-long consistent video chat experiences.
  arXiv  Detail & Related papers  (2025-09-25T20:53:27Z)
• SkyReels-Audio: Omni Audio-Conditioned Talking Portraits in Video Diffusion Transformers [25.36460340267922]
  We present SkyReels-Audio, a unified framework for synthesizing high-fidelity and temporally coherent talking portrait videos.<n>Our framework supports infinite-length generation and editing, while enabling diverse and controllable conditioning through multimodal inputs.
  arXiv  Detail & Related papers  (2025-06-01T04:27:13Z)
• OmniResponse: Online Multimodal Conversational Response Generation in Dyadic Interactions [62.19092662469285]
  Online Multimodal Conversational Response Generation (OMCRG) is a novel task designed to produce synchronized verbal and non-verbal listener feedback online.<n>We propose OmniResponse, a Multimodal Large Language Model (MLLM) that autoregressively generates accurate multimodal listener responses.<n>We offer ResponseNet, a dataset of 696 detailed dyadic interactions featuring synchronized split-screen videos, multichannel audio, transcripts, and annotated facial behaviors.
  arXiv  Detail & Related papers  (2025-05-27T20:12:46Z)
• AsynFusion: Towards Asynchronous Latent Consistency Models for Decoupled Whole-Body Audio-Driven Avatars [71.90109867684025]
  Whole-body audio-driven avatar pose and expression generation is a critical task for creating lifelike digital humans.<n>We propose AsynFusion, a novel framework that leverages diffusion transformers to achieve cohesive expression and gesture synthesis.<n>AsynFusion achieves state-of-the-art performance in generating real-time, synchronized whole-body animations.
  arXiv  Detail & Related papers  (2025-05-21T03:28:53Z)
• OmniTalker: One-shot Real-time Text-Driven Talking Audio-Video Generation With Multimodal Style Mimicking [22.337906095079198]
  We present OmniTalker, a unified framework that jointly generates synchronized talking audio-video content from input text.<n>Our framework adopts a dual-branch diffusion transformer (DiT) architecture, with one branch dedicated to audio generation and the other to video synthesis.
  arXiv  Detail & Related papers  (2025-04-03T09:48:13Z)
• Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animation [29.87407471246318]
  This research delves into the complexities of synchronizing facial movements and creating visually appealing, temporally consistent animations. Our innovative approach embraces the end-to-end diffusion paradigm and introduces a hierarchical audio-driven visual synthesis module. The proposed hierarchical audio-driven visual synthesis offers adaptive control over expression and pose diversity, enabling more effective personalization tailored to different identities.
  arXiv  Detail & Related papers  (2024-06-13T04:33:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.