Fugu-MT: arxivの論文翻訳
Daily Report / 2026-06-03

Daily Paper Briefing

2026-06-03 1 Paper

Audio Interaction Model

Authors Zhifei Xie, Zihang Liu, Ze An, Xiaobin Hu, Yue Liao, Ziyang Ma, Dongchao Yang, Mingbao Lin, Deheng Ye, Shuicheng Yan, Chunyan Miao
Affiliations Nanyang Technological University / The Chinese University of Hong Kong / National University of Singapore
Categories Method / Audio Interaction Model / Real-time perceptual-cognitive loop, Application / Streaming Audio Processing / Integrated streaming and offline task execution, Task / Speech and Audio Tasks / ASR and voice chat single-task handling
License CC BY 4.0

Abstract Overview

This paper formalizes the Audio Interaction Model, a streaming audio-language setting in which a model continuously listens to audio and decides when to remain silent or respond. The proposed system, Audio-Interaction, is implemented through the SoundFlow framework, which combines streaming-native data construction, chunk-level training for response triggering, and asynchronous FIFO-based inference for low-latency deployment. To support this regime, the authors build StreamAudio-2M, a 2.6M-item, 302k-hour corpus covering 7 major capability categories and 28 sub-tasks, and introduce Proactive-Sound-Bench for proactive audio intervention. The paper argues that this setup unifies conventional offline audio tasks with streaming-native behaviors such as real-time ASR, audio instruction following, dialogue, translation, and proactive assistance.

Novelty

The main novelty is the shift from offline large audio language models or single-task streaming systems to a unified, always-on perceive-decide-respond audio interaction model. The work also contributes an end-to-end streaming framework (SoundFlow), a large streaming-native dataset, and a benchmark specifically designed to test proactive audio-triggered responses.

Results

Across 8 benchmarks, Audio-Interaction preserves competitive performance on mainstream audio tasks while enabling streaming capabilities that offline systems do not express. It reaches 58.15 on MMAU under audio instructions versus 57.81 for its Qwen2.5-Omni-3B initialization, achieves 55.22/35.21 BLEU on CoVoST2 en-zh/zh-en, and scores 61.2/62.8 on the Single/Multi tiers of Proactive-Sound-Bench. The FIFO asynchronous inference design also reduces first-frame latency for resuming listening, with the paper reporting 392 ms average first-chunk latency versus 831 ms without FIFO.

Key Points

  1. Audio-Interaction reframes audio modeling as continuous chunk-wise interaction, where the model explicitly predicts whether to stay silent or generate a response.
  2. The paper introduces supporting infrastructure for this setting, including the SoundFlow training/inference pipeline, the 2.6M-item StreamAudio-2M corpus, and Proactive-Sound-Bench.
  3. Empirically, the model retains competitive general audio and speech performance while adding real-time instruction following, proactive response, and longer-stream robustness.

References

Hugging Face Project
This page was created using generative AI such as GPT-5, Claude Opus 4, Gemini 3, Gemini 3.1 Flash Image, and their higher-end successor versions. No guarantee can be made regarding its contents.