A Voice-Interactive Multi-Agent System for Smart Operating Rooms: Architecture Design and Key Technologies

• URL: http://arxiv.org/abs/2609.11231v2
• Date: Mon, 14 Sep 2026 03:32:00 GMT
• Title: A Voice-Interactive Multi-Agent System for Smart Operating Rooms: Architecture Design and Key Technologies
• Abstract summary: SurgicalRoomAgent is a voice-interactive multi-agent system for smart operating rooms based on large language models (LLMs)<n>System achieves natural language understanding, device control, intraoperative recording, and surgical report generation through a voice interaction pipeline.
• Score: 0.2538209532048867
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper presents SurgicalRoomAgent, a voice-interactive multi-agent system for smart operating rooms based on large language models (LLMs). The system achieves natural language understanding, device control, intraoperative recording, and surgical report generation through a layered architecture comprising a voice interaction pipeline (wake, ASR, turn detection, agent reasoning, TTS) and an agent core (skill registry, task planner, device manager). Three key technologies are investigated: (1) KV Cache prefix warming for low-latency inference, reducing recomputation overhead from approximately 500 ms to tens of milliseconds via byte-level Longest Common Prefix reuse; (2) streaming partial JSON parsing with early parallel task execution, reducing end-to-end latency by approximately 30%; and (3) progressive skill prompt disclosure, which dynamically filters system prompts based on user role, connected devices, and surgical phase to maximize information density within limited context windows. The system is implemented using the Qwen3-27B model with llama.cpp/sglang inference engines. Experimental analysis demonstrates effective operation within a 16,384-token context limit and multi-device parallel control response times meeting OR real-time requirements.

Related papers

• Speculative Interaction Agents: Building Real-Time Agents with Asynchronous I/O and Speculative Tool Calling [64.40340291543971]
  We propose Speculative Interaction Agents to enable real-time interaction with small edge-scale models.<n>We also present a clock-based training methodology that adapts the model to handle streaming inputs and asynchronous responses.<n>This approach provides 1.6-2.2$times$ speedups with the Qwen2.5-3B-Instruct and Llama-3.2-3B-Instruct models across multiple tool calling benchmarks.
  arXiv  Detail & Related papers  (2026-05-13T11:20:52Z)
• Audio2Tool: Speak, Call, Act -- A Dataset for Benchmarking Speech Tool Use [1.851890212523342]
  We introduce Audio2Tool to assess tool-calling capabilities of SpeechLMs across three primary domains: Smart Car, Smart Home, and Wearables.<n>Our benchmark features a multi-tier complexity hierarchy, ranging from simple direct commands to complex multi-intent and needle-in-a-haystack extraction to isolate distinct failure modes.<n> Evaluations of state-of-the-art SpeechLMs and ASR-LLM pipelines show strong performance on simple commands but significant degradation under compositional and acoustic challenges.
  arXiv  Detail & Related papers  (2026-04-17T16:41:25Z)
• AgentCgroup: Understanding and Controlling OS Resources of AI Agents [2.8139711959925244]
  AI agents are increasingly deployed in multi-tenant cloud environments, where they execute diverse tool calls within sandboxed containers.<n>We present a systematic characterization of OS-level resource dynamics in sandboxed AI coding agents.<n>Preliminary evaluation demonstrates improved multi-tenant isolation and reduced resource waste.
  arXiv  Detail & Related papers  (2026-02-10T02:37:42Z)
• LTS-VoiceAgent: A Listen-Think-Speak Framework for Efficient Streaming Voice Interaction via Semantic Triggering and Incremental Reasoning [27.13598270494417]
  LTS-VoiceAgent is a Listen-Think-Speak framework that separates when to think from how to reason incrementally.<n>It features a Dynamic Semantic Trigger to detect meaningful prefixes, and a Dual-Role Stream Orchestrator that coordinates a background Thinker and a foreground Speaker.
  arXiv  Detail & Related papers  (2026-01-26T15:42:35Z)
• Qwen3-TTS Technical Report [64.94647392030824]
  We present the Qwen3-TTS series, a family of advanced multilingual, controllable, robust, and streaming text-to-speech models.<n>Qwen3-TTS supports state-of-the-art 3-second voice cloning and description-based control.<n>Qwen3-TTS adopts a dual-track LM architecture for real-time synthesis, coupled with two speech tokenizers.
  arXiv  Detail & Related papers  (2026-01-22T03:51:43Z)
• AsyncVoice Agent: Real-Time Explanation for LLM Planning and Reasoning [27.522862635055077]
  We present AsyncVoice Agent, a system whose asynchronous architecture decouples a streaming backend from a conversational voice.<n>This design allows narration and inference to run in parallel, empowering users to interrupt, query, and steer the model's reasoning process.<n>Objective benchmarks show this approach reduces interaction latency by more than 600x compared to monolithic baselines.
  arXiv  Detail & Related papers  (2025-10-17T19:00:08Z)
• Stream RAG: Instant and Accurate Spoken Dialogue Systems with Streaming Tool Usage [66.67531241554546]
  End-to-end speech-in speech-out dialogue systems are emerging as a powerful alternative to traditional ASR-LLM-TTS pipelines.<n>We introduce the first approach to extend tool use directly into speech-in speech-out systems.<n>We propose Streaming Retrieval-Augmented Generation (Streaming RAG), a novel framework that reduces user-perceived latency by predicting tool queries in parallel with user speech.
  arXiv  Detail & Related papers  (2025-10-02T14:18:20Z)
• InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactions [104.90258030688256]
  This project introduces disentangled streaming perception, reasoning, and memory mechanisms, enabling real-time interaction with streaming video and audio input.<n>This project simulates human-like cognition, enabling multimodal large language models to provide continuous and adaptive service over time.
  arXiv  Detail & Related papers  (2024-12-12T18:58:30Z)
• Asynchronous Tool Usage for Real-Time Agents [61.3041983544042]
  We introduce asynchronous AI agents capable of parallel processing and real-time tool-use. Our key contribution is an event-driven finite-state machine architecture for agent execution and prompting. This work presents both a conceptual framework and practical tools for creating AI agents capable of fluid, multitasking interactions.
  arXiv  Detail & Related papers  (2024-10-28T23:57:19Z)
• Multi-modal Queried Object Detection in the Wild [72.16067634379226]
  MQ-Det is an efficient architecture and pre-training strategy design for real-world object detection. It incorporates vision queries into existing language-queried-only detectors. MQ-Det's simple yet effective architecture and training strategy design is compatible with most language-queried object detectors.
  arXiv  Detail & Related papers  (2023-05-30T12:24:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.