Aero Realtime: Fully Aligned Input-Output Streams for Low-Latency Streaming Multimodal Generation
- URL: http://arxiv.org/abs/2608.08469v1
- Date: Sun, 09 Aug 2026 04:22:57 GMT
- Title: Aero Realtime: Fully Aligned Input-Output Streams for Low-Latency Streaming Multimodal Generation
- Abstract summary: We introduce Aero Realtime, which aligns video, audio and textual output on a shared temporal grid for realtime generation.<n>On four NVIDIA A6000 workstations, Aero Realtime maintains 84-ms median and 173-ms P95 processing lag over 20 minutes of a continuously streamed video.<n>Results demonstrate the feasibility of fully input-output modeling for duplex proactive and hardware-aligned multimodal interaction.
- Score: 33.4073718746898
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Existing streaming multimodal models process observations incrementally but still follow a turn-based prefill-then-decode pattern, making them non-duplex: new observations cannot naturally enter an active generation stream. Proactive alternatives use micro-turn polling or external response gates, which fragment continuous interaction, decouple response timing from language generation, and complicate KV-cache-friendly serving. We introduce Aero Realtime, a 4B streaming multimodal model with a duplex architecture for realtime generation. Aero Realtime aligns video, audio, and textual output on a shared temporal grid, where each approximately 80-ms audio slot predicts either a lexical token or a silence token. This allows input and output to advance together, enabling one autoregressive objective to learn both when to respond and what to generate. During inference, Aero Realtime appends only the newest multimodal slot, carries forward the previous output state, and reuses the KV cache for efficient incremental execution. We further provide a complete training and serving recipe, including realtime QA construction, slot-aligned supervision, hardware-aware distributed training, and resumable inference. On four NVIDIA A6000 workstation GPUs, Aero Realtime maintains 84-ms median and 173-ms P95 processing lag over 20 minutes of a continuously streamed video, remaining within 200~ms of the source timeline. These results demonstrate the feasibility of fully aligned input-output modeling for duplex, proactive, and hardware-aligned multimodal interaction.
Related papers
- StreamPI: Streaming Multimodal Temporal Modeling for Vision-Language-Action Models [84.17128030384099]
StreamPI is a streaming multimodal temporal modeling framework.<n>It equips single-frame VLA with temporal reasoning capability without introducing any additional parameters.<n> Experiments on real-robot tasks spanning memory-dependent and precise perception scenarios, as well as the simulation benchmark LIBERO, demonstrate that StreamPI outperforms pi0.5 across diverse tasks.
arXiv Detail & Related papers (2026-08-26T17:33:19Z) - StreamArena: Toward Continuous, Interactive, and Long-Horizon Agentic Streaming Video Understanding [42.88155526332792]
We introduce StreamArena, a benchmark for hour-scale, interactive streaming video understanding.<n> evaluation across diverse systems exposes a tension between continuous interaction and long-horizon multimodal comprehension.<n>We address this tension with StreamMind, a two-tier architecture that assigns latency-critical interaction and proactive monitoring to independently scheduled workers.
arXiv Detail & Related papers (2026-08-06T07:46:37Z) - Wan-Streamer v0.1: End-to-end Real-time Interactive Foundation Models [68.51256516936338]
Wan-Streamer is an end-to-end native interactive foundation model for low-streaming interaction.<n>It seamlessly models audio and video as both input and output within single Transformer sequence.<n>It achieves approximately 200 ms model-side response latency and approximately 550 ms total interaction latency.
arXiv Detail & Related papers (2026-06-23T18:01:03Z) - Don't Pause: Streaming Video-Language Synchrony for Online Video Understanding [69.296913137409]
We introduce a novel paradigm for online video understanding: Streaming Video-Language Synchrony (SVLS)<n>We present LyraV, a live streaming assistant built upon a hierarchical control framework with two core innovations.<n>First, the Frame-Driven Transition Controller (FDTC), makes high-level semantic decisions on when to continue speaking, start a new response, or stay silent.<n>Second, the Streaming Token Pacer (SToP), a plug-and-play lightweight predictive module, dynamically adapts the language generation rate to match the pace of the visual content.
arXiv Detail & Related papers (2026-06-05T07:29:20Z) - OmniInteract: Benchmarking Real-World Streaming Interaction for Real-Time Omnimodal Assistants [43.2907783462394]
Streaming benchmark for realtime omnimodal language models evaluated through native inference over audio-visual streams.<n>We evaluate response correctness, timing, invalid outputs interruption, and context continuity using Interaction-Aware QualityTimeliness F1, Interruption Diagnostic Suite, and 368 1QnA slots for continuous task monitoring and step guidance.
arXiv Detail & Related papers (2026-05-26T02:49:47Z) - MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction [76.4461698685681]
Mini-o 4.5 is our latest effort towards human-level realtime streaming interaction.<n>It is based on Omni-CPM, a unified streaming framework that aligns omni-modal perception and outputs along a shared temporal axis.<n>With a total of 9B parameters, Mini-o 4.5 approaches Gemini 2.5 Flash in vision computation capabilities, delivering stateofthe-art open computation performance scale.
arXiv Detail & Related papers (2026-04-30T04:05:43Z) - StreamDiffusionV2: A Streaming System for Dynamic and Interactive Video Generation [65.90400162290057]
Generative models are reshaping the live-streaming industry by redefining how content is created, styled, and delivered.<n>Recent advances in video diffusion have markedly improved temporal consistency and sampling efficiency for offline generation.<n>Live online streaming operates under strict service-level objectives (SLOs): time-to-first-frame must be minimal, and every frame must meet a per-frame deadline with low jitter.
arXiv Detail & Related papers (2025-11-10T18:51:28Z) - STORM: Token-Efficient Long Video Understanding for Multimodal LLMs [116.4479155699528]
STORM is a novel architecture incorporating a dedicated temporal encoder between the image encoder and the Video-LLMs.<n>We show that STORM achieves state-of-the-art results across various long video understanding benchmarks.
arXiv Detail & Related papers (2025-03-06T06:17:38Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.