Related papers: Realtime-Venus: A full-duplex interaction system with asynchronous delegation

Realtime-Venus: A full-duplex interaction system with asynchronous delegation

URL: http://arxiv.org/abs/2609.13814v1
Date: Sat, 12 Sep 2026 08:56:50 GMT
Title: Realtime-Venus: A full-duplex interaction system with asynchronous delegation
Abstract summary: We present Realtime, Realtime Omni audio visual interaction and Realtime-Audio Spoken for spoken interaction.<n>Each model serves as complete conversational, integrating continuous perception, conversational control, and native speech generation.
Score: 59.69745588222872
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
Abstract: Natural interaction in digital and physical environments requires continuous perception and timely responses. Spoken dialogue relies on acoustic and linguistic cues, while video interaction also requires grounding the conversation in evolving visual context. We present Realtime-Venus, a proactive full-duplex interaction system with two separately trained 9B models: Realtime-Venus-Omni for audio-visual interaction and Realtime-Venus-Audio for spoken interaction. Each model serves as a complete conversational frontend, integrating continuous perception, conversational control, and native speech generation through a shared causal timeline for user inputs, model outputs, and delegation events. A dual-loop runtime coordinates live interaction with background reasoning and tool execution. Foreground interaction continues while Realtime-Venus-Harness executes tasks asynchronously and returns results for integration into the ongoing dialogue. Both models follow a common post-training recipe combining offline understanding, proactive full-duplex trajectories, and delegation workflows. Among the evaluated online models, Realtime-Venus-Omni achieves the highest scores on six of eight video benchmarks, including StreamingBench (70.2%), OVO-Bench (64.7%), and Daily-Omni (81.3%). Across eight audio understanding and spoken question answering benchmarks, Realtime-Venus-Audio leads the compared models on MMAU (78.0%), MMAU-Pro (63.2%), Llama Questions (83.8%), and Speech CMMLU (67.8%), while matching the best VoiceBench AlpacaEval score of 4.81. On Full-Duplex-Bench v1.5, Realtime-Venus-Audio responds to 75% of user interruptions and achieves continuation rates of 97%, 88%, and 86% under backchannels, other-directed speech, and background speech, respectively, exceeding Gemini 3.1 Live and GPT-4o on all three continuation metrics.

Related papers

Aero Realtime: Fully Aligned Input-Output Streams for Low-Latency Streaming Multimodal Generation [33.4073718746898]
We introduce Aero Realtime, which aligns video, audio and textual output on a shared temporal grid for realtime generation.<n>On four NVIDIA A6000 workstations, Aero Realtime maintains 84-ms median and 173-ms P95 processing lag over 20 minutes of a continuously streamed video.<n>Results demonstrate the feasibility of fully input-output modeling for duplex proactive and hardware-aligned multimodal interaction.
arXiv Detail & Related papers (2026-08-09T04:22:57Z)
Multi-Faceted Interactivity Alignment in Full-Duplex Speech Models [53.470209949659115]
We propose a post-training alignment method that improves full spoken dialogue models through RL.<n>We address the four canonical axes of interactivity: handling turn-taking, back-channel pause, and user interruption.<n>An extra-based reward for response quality prevents semantic degradation.
arXiv Detail & Related papers (2026-06-09T17:46:55Z)
Audio Interaction Model [102.4354125819644]
Today's Large Audio Language Models (LALMs) are offline, and streaming audio models each handle only a single task such as streaming ASR or voice chatting.<n>It is time to unify them into one online LALM: a model that, through an always-on perceive-decide-respond loop, listens to sound, environment, and instructions in real time and reacts on the fly.<n>We formalize this regime as the Audio Interaction Model, and realize it with Audio-Interaction, a unified streaming model that retains offline task execution while adding online general audio instruction following.
arXiv Detail & Related papers (2026-06-03T17:26:11Z)
OmniInteract: Benchmarking Real-World Streaming Interaction for Real-Time Omnimodal Assistants [43.2907783462394]
Streaming benchmark for realtime omnimodal language models evaluated through native inference over audio-visual streams.<n>We evaluate response correctness, timing, invalid outputs interruption, and context continuity using Interaction-Aware QualityTimeliness F1, Interruption Diagnostic Suite, and 368 1QnA slots for continuous task monitoring and step guidance.
arXiv Detail & Related papers (2026-05-26T02:49:47Z)
MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction [76.4461698685681]
Mini-o 4.5 is our latest effort towards human-level realtime streaming interaction.<n>It is based on Omni-CPM, a unified streaming framework that aligns omni-modal perception and outputs along a shared temporal axis.<n>With a total of 9B parameters, Mini-o 4.5 approaches Gemini 2.5 Flash in vision computation capabilities, delivering stateofthe-art open computation performance scale.
arXiv Detail & Related papers (2026-04-30T04:05:43Z)
Full-Duplex-Bench-v3: Benchmarking Tool Use for Full-Duplex Voice Agents Under Real-World Disfluency [61.68376148916503]
We introduce Full-Duplex-Bench-v3 (FDB-v3), a benchmark for evaluating spoken language models under naturalistic speech conditions and multi-step tool use.<n>Unlike prior work, our dataset consists entirely of real human audio annotated for five disfluency categories, paired with scenarios requiring chained API calls across four task domains.<n>We evaluate six model configurations -- GPT-Realtime, Gemini Live 2.5, Gemini Live 3.1, Grok, Ultravox v0.7, and a traditional Cascaded pipeline (Whisper$rightarrow$rightarrow$TTS) -- across accuracy, latency, and turn-
arXiv Detail & Related papers (2026-04-06T16:46:52Z)
Real-Time Voicemail Detection in Telephony Audio Using Temporal Speech Activity Features [1.066048003460524]
We present a lightweight approach that extracts 15 temporal features from the speech activity pattern of a pre-trained neural voice activity detector.<n>Across two evaluation sets totaling 764 telephony recordings, the system achieves a combined 96.1% accuracy.
arXiv Detail & Related papers (2026-04-02T17:43:24Z)
Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities [93.09944267871163]
FullDuplexBench is a benchmark that systematically evaluates key interactive behaviors.<n>By releasing our benchmark code we aim to advance spoken dialogue modeling and the development of more natural and engaging SDMs.
arXiv Detail & Related papers (2025-03-06T18:59:16Z)
Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agents [12.555910887280199]
We propose Synchronous LLMs for full-sync spoken dialogue modeling. We train a model that generates meaningful and natural spoken dialogue with just 2k hours of real-world spoken dialogue data. We demonstrate the model's ability to participate in full-sync dialogue by simulating interaction between two agents trained on different datasets.
arXiv Detail & Related papers (2024-09-23T23:01:31Z)
Real-time Speaker counting in a cocktail party scenario using Attention-guided Convolutional Neural Network [60.99112031408449]
We propose a real-time, single-channel attention-guided Convolutional Neural Network (CNN) to estimate the number of active speakers in overlapping speech. The proposed system extracts higher-level information from the speech spectral content using a CNN model. Experiments on simulated overlapping speech using WSJ corpus show that the attention solution is shown to improve the performance by almost 3% absolute over conventional temporal average pooling.
arXiv Detail & Related papers (2021-10-30T19:24:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.