Speculative Interaction Agents: Building Real-Time Agents with Asynchronous I/O and Speculative Tool Calling

• URL: http://arxiv.org/abs/2605.13360v2
• Date: Thu, 14 May 2026 17:52:40 GMT
• Title: Speculative Interaction Agents: Building Real-Time Agents with Asynchronous I/O and Speculative Tool Calling
• Abstract summary: We propose Speculative Interaction Agents to enable real-time interaction with small edge-scale models.<n>We also present a clock-based training methodology that adapts the model to handle streaming inputs and asynchronous responses.<n>This approach provides 1.6-2.2$times$ speedups with the Qwen2.5-3B-Instruct and Llama-3.2-3B-Instruct models across multiple tool calling benchmarks.
• Score: 64.40340291543971
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: There is a growing demand for agentic AI technologies for a range of downstream applications like customer service and personal assistants. For applications where the agent needs to interact with a person, real-time low-latency responsiveness is required; for example, with voice-controlled applications, under 1 second of latency is typically required for the interaction to feel seamless. However, if we want the LLM to reason and execute an agentic workflow with tool calling, this can add several seconds or more of latency, which is prohibitive for real-time latency-sensitive applications. In our work, we propose Speculative Interaction Agents to enable real-time interaction even for agents with complex multi-turn tool calling. We propose Asynchronous I/O, which decouples the core agent reason-and-act thread from waiting for additional information from either the user or environment, thereby allowing for overlapping agentic processing while waiting on external delays. We also propose Speculative Tool Calling as a method to manage task execution when the agent is still unsure if it has received the full information or if additional user information may later be provided. For strong cloud models, our method can be applied out-of-the-box to existing real-time cloud APIs, providing 1.3-1.7$\times$ speedups with minor accuracy loss. To enable real-time interaction with small edge-scale models, we also present a clock-based training methodology that adapts the model to handle streaming inputs and asynchronous responses, and demonstrate a synthetic data generation strategy for SFT. Altogether, this approach provides 1.6-2.2$\times$ speedups with the Qwen2.5-3B-Instruct and Llama-3.2-3B-Instruct models across multiple tool calling benchmarks.

Related papers

• Coordinated Networking for On-Device Agent-Augmented Real-Time Communication [24.853091472152084]
  We design a framework to ensure both high live video quality and low agent response latency in agent-augmented RTC apps.<n>Our prototype built atop and llama demonstrates that HAFS outperforms baselines, achieving 1.5x higher video quality while reducing agent response time by 31%.
  arXiv  Detail & Related papers  (2026-07-24T18:52:41Z)
• AsyncTool: Evaluating the Asynchronous Function Calling Capability under Multi-Task Scenarios [33.202154532105396]
  Large language model (LLM)-based agents have shown strong capabilities in using external tools to solve complex tasks.<n>In real-world applications, multiple tasks often need to be executed concurrently, and overall efficiency depends on whether an agent can use idle time while waiting for tool responses.<n>We propose AsyncTool, a benchmark for assessing LLM-based agents in interactive multi-task tool-use environments with delayed tool feedback.
  arXiv  Detail & Related papers  (2026-05-27T05:36:35Z)
• SAGA: Workflow-Atomic Scheduling for AI Agent Inference on GPU Clusters [13.891522069967507]
  We propose a shift to program-level scheduling for compound AI workloads.<n>We present SAGA, a distributed scheduler that implements this abstraction through three mechanisms.
  arXiv  Detail & Related papers  (2026-05-01T09:05:28Z)
• MindWatcher: Toward Smarter Multimodal Tool-Integrated Reasoning [55.221850286246]
  We introduce MindWatcher, a tool-integrated reasoning agent with interleaved thinking and multimodal chain-of-thought (CoT) reasoning.<n>MindWatcher can autonomously decide whether and how to invoke diverse tools and coordinate their use.<n>A large-scale, high-quality local image retrieval database, covering eight categories including cars, animals, and plants, endows model with robust object recognition.
  arXiv  Detail & Related papers  (2025-12-29T12:16:12Z)
• Towards Efficient Agents: A Co-Design of Inference Architecture and System [66.59916327634639]
  This paper presents AgentInfer, a unified framework for end-to-end agent acceleration.<n>We decompose the problem into four synergistic components: AgentCollab, AgentSched, AgentSAM, and AgentCompress.<n>Experiments on the BrowseComp-zh and DeepDiver benchmarks demonstrate that through the synergistic collaboration of these methods, AgentInfer reduces ineffective token consumption by over 50%.
  arXiv  Detail & Related papers  (2025-12-20T12:06:13Z)
• DeepAgent: A General Reasoning Agent with Scalable Toolsets [111.6384541877723]
  DeepAgent is an end-to-end deep reasoning agent that performs autonomous thinking, tool discovery, and action execution.<n>To address the challenges of long-horizon interactions, we introduce an autonomous memory folding mechanism that compresses past interactions into structured episodic, working, and tool memories.<n>We develop an end-to-end reinforcement learning strategy, namely ToolPO, that leverages LLM-simulated APIs and applies tool-call advantage attribution to assign fine-grained credit to the tool invocation tokens.
  arXiv  Detail & Related papers  (2025-10-24T16:24:01Z)
• AsyncVoice Agent: Real-Time Explanation for LLM Planning and Reasoning [27.522862635055077]
  We present AsyncVoice Agent, a system whose asynchronous architecture decouples a streaming backend from a conversational voice.<n>This design allows narration and inference to run in parallel, empowering users to interrupt, query, and steer the model's reasoning process.<n>Objective benchmarks show this approach reduces interaction latency by more than 600x compared to monolithic baselines.
  arXiv  Detail & Related papers  (2025-10-17T19:00:08Z)
• Asynchronous Tool Usage for Real-Time Agents [61.3041983544042]
  We introduce asynchronous AI agents capable of parallel processing and real-time tool-use. Our key contribution is an event-driven finite-state machine architecture for agent execution and prompting. This work presents both a conceptual framework and practical tools for creating AI agents capable of fluid, multitasking interactions.
  arXiv  Detail & Related papers  (2024-10-28T23:57:19Z)
• Octopus v2: On-device language model for super agent [10.998608318944985]
  Our research presents a new method that empowers an on-device model with 2 billion parameters to surpass the performance of GPT-4 in both accuracy and latency. When compared to Llama-7B with a RAG-based function calling mechanism, our method enhances latency by 35-fold.
  arXiv  Detail & Related papers  (2024-04-02T09:01:32Z)
• AgentScope: A Flexible yet Robust Multi-Agent Platform [66.64116117163755]
  AgentScope is a developer-centric multi-agent platform with message exchange as its core communication mechanism. The abundant syntactic tools, built-in agents and service functions, user-friendly interfaces for application demonstration and utility monitor, zero-code programming workstation, and automatic prompt tuning mechanism significantly lower the barriers to both development and deployment.
  arXiv  Detail & Related papers  (2024-02-21T04:11:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.