AOSpec: Action and Observation Co-Speculation for Low-Latency Agent Serving

• URL: http://arxiv.org/abs/2608.00881v1
• Date: Sat, 01 Aug 2026 22:06:52 GMT
• Title: AOSpec: Action and Observation Co-Speculation for Low-Latency Agent Serving
• Abstract summary: We present AOSpec, a framework that co-speculates actions and observations across the full agent-environment loop.<n>For outcomes only execution can reveal, AOSpec launches latency-critical target actions in isolated forks.<n>We show that AOSpec reduces mean end-to-end latency by 11.8-32.5% and p99 latency by up to 42.8%.
• Score: 12.403374013868019
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language model agents increasingly act through stateful tools, yet model generation and environment execution remain serialized at every step. As decoding accelerates, tool execution becomes a growing bottleneck. Existing action- or observation-only speculation leaves much of this latency exposed: value is concentrated in a few slow calls, some outcomes emerge only through execution, and longer lookahead typically requires an increasingly unlikely chain of action predictions. We present AOSpec, a lossless framework that co-speculates actions and observations across the full agent-environment loop. Expected Value Decoding (EVD) directs observation speculation toward outcomes with the greatest expected latency benefit, optimizing expected time hidden rather than hit rate. For outcomes only execution can reveal, AOSpec launches latency-critical target actions in isolated forks that contain their effects, while Joint Action-State Verification (JASV) verifies both the action and its origin state against committed execution before reuse. JASV recasts long-horizon action dependency from full-chain prediction into target action-state verification, breaking the lookahead--accuracy tradeoff and unlocking long-range overlap without sacrificing serial semantics. Across Terminal-Bench serving settings spanning four harnesses, five actor models, and five serving speeds, AOSpec outperforms every practical baseline, reducing mean end-to-end latency by 11.8-32.5% and p99 latency by up to 42.8%. Its gains increase as decoding accelerates, and its observation model transfers from Terminal-Bench to SWE-bench Verified without retraining.

Related papers

• Speculative Macro Commit for Faster Tool-Using Agents [18.574702996656935]
  We introduce textbfSpeculative Macro Commit (SMC), a runtime mechanism for a two-tier agent system.<n>SMC mines recurring multi-action skeletons from training traces and stores them in a macro library.<n>On AppWorld, SMC reduces wall time by 7.7% over SA baseline and 44.9% over sequential execution.
  arXiv  Detail & Related papers  (2026-09-03T00:31:43Z)
• Latent Action as Intention Enables Efficient Future Imagination for World Action Models [20.031922437200524]
  We introduce **LAWA**, a WAM architecture that uses compact latent actions as an operational representation of future intentions.<n>On RoboCasa, LAWA achieves state-of-the-art average success rates of 65.6% and 80.8% in the few-shot and full data settings.<n>It also preserves the performance level of the matched Joint-WAM variant while requiring 42.9% lower inference latency.
  arXiv  Detail & Related papers  (2026-08-25T17:59:03Z)
• FutureRTC: Real-Time Robot Execution with Anticipatory-Conditioned Action Chunking [51.97839311685636]
  FutureRTC is a plug-and-play adaptation framework that predicts execution-time observations and states for asynchronous VLA control without modifying the underlying policy.<n>We show that FutureRTC achieves robustness to inference delays, resulting in smoother trajectories, faster execution, and consistently higher task success rates.
  arXiv  Detail & Related papers  (2026-07-27T05:10:16Z)
• IdleSpec: Exploiting Idle Time via Speculative Planning for LLM Agents [56.77101184011525]
  Large language model (LLM)-based agents solve complex tasks by leveraging multi-step reasoning with iterative tool calls and environment interactions.<n>Despite the prevalence of idle time in most agentic scenarios, existing works treat it as an unavoidable overhead.<n>We introduce IdleSpec, a scalable and generic inference approach that leverages idle-time computation to improve agent performance.
  arXiv  Detail & Related papers  (2026-05-21T08:25:17Z)
• SpecHop: Continuous Speculation for Accelerating Multi-Hop Retrieval Agents [55.74932252501046]
  SpecHop is a continuous speculation framework that maintains multiple speculative threads.<n>We show that SpecHop can approach oracle latency gains with enough active threads.
  arXiv  Detail & Related papers  (2026-05-21T03:55:47Z)
• SpecEyes: Accelerating Agentic Multimodal LLMs via Speculative Perception and Planning [104.01865949020304]
  Agentic multimodal large language models (MLLMs) achieve remarkable reasoning capabilities through iterative visual tool invocation.<n>However, the cascaded perception, reasoning, and tool-calling loops introduce significant sequential overhead.<n>This overhead, termed agentic depth, incurs prohibitive latency and seriously limits system-level throughput.<n>We propose SpecEyes, an agentic-level speculative acceleration framework that breaks this sequential bottleneck.
  arXiv  Detail & Related papers  (2026-03-24T17:45:47Z)
• FASTER: Rethinking Real-Time Flow VLAs [82.58822112377923]
  Real-time execution is crucial for deploying Vision-Language-Action (VLA) models in the physical world.<n>We show that reaction time follows a uniform distribution determined jointly by the Time to First Action (TTFA) and the execution horizon.<n>We propose Fast Action Sampling for ImmediaTE Reaction (FASTER) to overcome this issue.
  arXiv  Detail & Related papers  (2026-03-19T17:51:37Z)
• Reducing Latency of LLM Search Agent via Speculation-based Algorithm-System Co-Design [35.95362310928356]
  LLM-based search agents achieve strong performance but suffer from severe latency.<n>We revisit this bottleneck through the lens of speculation.<n>We present SPAgent, an algorithm-system co-design framework that expands the role of speculation in search agents to reduce latency.
  arXiv  Detail & Related papers  (2025-11-25T08:15:17Z)
• SpecExit: Accelerating Large Reasoning Model via Speculative Exit [10.522333173441453]
  We propose SpecExit, a framework that predicts both future tokens and an early-exit signal directly from a draft model without probing overhead.<n>Our method offers significant improvements, reducing average generation length by 66% and achieving a 2.5x speedup in end-to-end latency.
  arXiv  Detail & Related papers  (2025-09-29T03:39:32Z)
• LSTC: Boosting Atomic Action Detection with Long-Short-Term Context [60.60267767456306]
  We decompose the action recognition pipeline into short-term and long-term reliance. Within our design, a local aggregation branch is utilized to gather dense and informative short-term cues. Both branches independently predict the context-specific actions and the results are merged in the end.
  arXiv  Detail & Related papers  (2021-10-19T10:09:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.