Related papers: Wan-Streamer v0.1: End-to-end Real-time Interactive Foundation Models

Wan-Streamer v0.1: End-to-end Real-time Interactive Foundation Models

URL: http://arxiv.org/abs/2606.25041v3
Date: Mon, 29 Jun 2026 05:06:50 GMT
Title: Wan-Streamer v0.1: End-to-end Real-time Interactive Foundation Models
Abstract summary: Wan-Streamer is an end-to-end native interactive foundation model for low-streaming interaction.<n>It seamlessly models audio and video as both input and output within single Transformer sequence.<n>It achieves approximately 200 ms model-side response latency and approximately 550 ms total interaction latency.
Score: 68.51256516936338
License: http://creativecommons.org/licenses/by/4.0/
Abstract: We present Wan-Streamer, a native-streaming, end-to-end interactive foundation model designed from the ground up for real-time, low-latency, full-duplex audio-visual interaction. Wan-Streamer seamlessly models language, audio, and video as both input and output within a single Transformer, where the sequence is represented as interleaved visual, audio, and text input tokens together with visual, audio, and text output tokens, coordinated by block-causal attention for incremental streaming. Unlike cascaded interactive systems that rely on separate VAD, ASR, language, TTS, audio-driven animation, or video-generation modules, Wan-Streamer does not rely on external language, speech, avatar, or video-generation modules: perception, reasoning, generation, response timing, turn management, and cross-modal synchronization are learned jointly within one unified model, reducing pipeline latency and error accumulation. To support natural audio-visual responsiveness, we redesign the entire stack around streamability, including causal encoders, causal decoders, block-causal attention, and low-latency multimodal token scheduling, enabling streaming units as short as 160 ms at 25 fps. Wan-Streamer achieves approximately 200 ms model-side response latency and approximately 550 ms total interaction latency when combined with 350 ms bidirectional network latency, supporting sub-second duplex audio-visual communication. These results position Wan-Streamer as a unified, end-to-end, multimodal interactive foundation model for low-latency streaming interaction.

Related papers

TontaubeV1: Streaming Text-to-Speech with Hierarchical Codec Modeling and Bounded Context [0.04583541422554719]
TontaubeV1 is a model that preserves natural prosody while enabling streaming from a single consumer GPU.<n>Our design assumes that prosodic structure is largely established when the semantic stream is generated.<n>The model accepts up to one minute of reference audio for voice conditioning and is designed primarily for English and German.
arXiv Detail & Related papers (2026-09-08T13:03:46Z)
Aero Realtime: Fully Aligned Input-Output Streams for Low-Latency Streaming Multimodal Generation [33.4073718746898]
We introduce Aero Realtime, which aligns video, audio and textual output on a shared temporal grid for realtime generation.<n>On four NVIDIA A6000 workstations, Aero Realtime maintains 84-ms median and 173-ms P95 processing lag over 20 minutes of a continuously streamed video.<n>Results demonstrate the feasibility of fully input-output modeling for duplex proactive and hardware-aligned multimodal interaction.
arXiv Detail & Related papers (2026-08-09T04:22:57Z)
Video = World + Event Stream [70.56197496608006]
Wan-Streamer v0.3 reframes our native interaction model under a single organizing: a is a world plus an event stream.<n>This yields a general-purpose pretraining task over large amounts of real video.<n> Functionally, the model's multimodal understanding process is vision--action-like.
arXiv Detail & Related papers (2026-07-16T14:19:42Z)
Wan-Streamer v0.2: Higher Resolution, Same Latency [68.78755383327113]
Wan-Streamer v0.2 is a latency-preserving upgrade of the native-streaming, end-to-end audio-visual interaction model.<n>It supports scene-grounded mid-shot agents whose posture, gaze, hands, nearby objects, and local scene layout remain legible during real-time conversation.
arXiv Detail & Related papers (2026-07-05T18:05:20Z)
An Efficient vLLM-Based Inference Pipeline for Unified Audio Understanding and Generation [63.50393485685279]
We present a vLLM-based inference pipeline for unified speech understanding and generation.<n>We overcome the shared intuition-Free Guidance (CFG) halves throughput.<n>By co-scheduling paired conditional and unconditional requests within a continuous batch, our CFG implementation sustains 80% of non-CFG throughput.
arXiv Detail & Related papers (2026-07-02T12:55:11Z)
Audio Interaction Model [102.4354125819644]
Today's Large Audio Language Models (LALMs) are offline, and streaming audio models each handle only a single task such as streaming ASR or voice chatting.<n>It is time to unify them into one online LALM: a model that, through an always-on perceive-decide-respond loop, listens to sound, environment, and instructions in real time and reacts on the fly.<n>We formalize this regime as the Audio Interaction Model, and realize it with Audio-Interaction, a unified streaming model that retains offline task execution while adding online general audio instruction following.
arXiv Detail & Related papers (2026-06-03T17:26:11Z)
OmniInteract: Benchmarking Real-World Streaming Interaction for Real-Time Omnimodal Assistants [43.2907783462394]
Streaming benchmark for realtime omnimodal language models evaluated through native inference over audio-visual streams.<n>We evaluate response correctness, timing, invalid outputs interruption, and context continuity using Interaction-Aware QualityTimeliness F1, Interruption Diagnostic Suite, and 368 1QnA slots for continuous task monitoring and step guidance.
arXiv Detail & Related papers (2026-05-26T02:49:47Z)
MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction [76.4461698685681]
Mini-o 4.5 is our latest effort towards human-level realtime streaming interaction.<n>It is based on Omni-CPM, a unified streaming framework that aligns omni-modal perception and outputs along a shared temporal axis.<n>With a total of 9B parameters, Mini-o 4.5 approaches Gemini 2.5 Flash in vision computation capabilities, delivering stateofthe-art open computation performance scale.
arXiv Detail & Related papers (2026-04-30T04:05:43Z)
Voxtral Realtime [134.66962524291424]
Voxtral Realtime is a streaming automatic speech recognition model.<n>It matches offline transcription quality at sub-second latency.<n>We release the model weights under the Apache 2.0 license.
arXiv Detail & Related papers (2026-02-11T19:17:10Z)
Real-Time Streamable Generative Speech Restoration with Flow Matching [35.33575179870606]
Stream$.$FM is a frame-causal flow-based generative model with an algorithmic latency of 32 milliseconds (ms) and a total latency of 48 ms.<n>We show that high-quality streaming generative speech processing can be realized on consumer GPUs available today.
arXiv Detail & Related papers (2025-12-22T14:41:17Z)
X-Streamer: Unified Human World Modeling with Audiovisual Interaction [36.50697656708077]
X-Streamer is a framework for building digital human agents capable of infinite interactions across text, speech, and video.<n>At its core is a Thinker-Actor dual-transformer architecture that unifies multimodal understanding and generation.<n>X-Streamer runs in real time on two A100 GPUs, sustaining hours-long consistent video chat experiences.
arXiv Detail & Related papers (2025-09-25T20:53:27Z)
VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Model [84.25283710008785]
VITA-Audio is an end-to-end large speech model with fast audio-text token generation.<n>MCTP module efficiently generates multiple audio tokens within a single model forward pass.<n>Four-stage progressive training strategy is explored to achieve model acceleration with minimal loss of speech quality.
arXiv Detail & Related papers (2025-05-06T17:59:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.