SpecHop: Continuous Speculation for Accelerating Multi-Hop Retrieval Agents

• URL: http://arxiv.org/abs/2605.21965v1
• Date: Thu, 21 May 2026 03:55:47 GMT
• Title: SpecHop: Continuous Speculation for Accelerating Multi-Hop Retrieval Agents
• Abstract summary: SpecHop is a continuous speculation framework that maintains multiple speculative threads.<n>We show that SpecHop can approach oracle latency gains with enough active threads.
• Score: 55.74932252501046
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models increasingly use external tools such as web search and document retrieval to solve information-intensive tasks. However, multi-hop tool use in complex tasks introduces substantial latency, since the model must repeatedly wait for tool observations before continuing. We study how to accelerate such trajectories without changing the final trajectory the model would have taken without acceleration, assuming access to faster but less reliable speculator tools. We develop a theoretical framework for lossless speculation in multi-hop tool-use settings, characterizing the optimal achievable latency gain. We propose SpecHop, a continuous speculation framework that maintains multiple speculative threads, verifies predicted observations asynchronously as target tool outputs arrive, commits correct branches, and rolls back incorrect ones. This preserves accuracy while reducing wall-clock latency. We show that SpecHop can approach oracle latency gains with enough active threads. Empirically, on retrieval-augmented multi-hop tasks, SpecHop closely matches theoretical predictions and reduces latency by up to 40\% in some settings. Code: https://github.com/mehrdadsaberi/spechop

Related papers

• AOSpec: Action and Observation Co-Speculation for Low-Latency Agent Serving [12.403374013868019]
  We present AOSpec, a framework that co-speculates actions and observations across the full agent-environment loop.<n>For outcomes only execution can reveal, AOSpec launches latency-critical target actions in isolated forks.<n>We show that AOSpec reduces mean end-to-end latency by 11.8-32.5% and p99 latency by up to 42.8%.
  arXiv  Detail & Related papers  (2026-08-01T22:06:52Z)
• IdleSpec: Exploiting Idle Time via Speculative Planning for LLM Agents [56.77101184011525]
  Large language model (LLM)-based agents solve complex tasks by leveraging multi-step reasoning with iterative tool calls and environment interactions.<n>Despite the prevalence of idle time in most agentic scenarios, existing works treat it as an unavoidable overhead.<n>We introduce IdleSpec, a scalable and generic inference approach that leverages idle-time computation to improve agent performance.
  arXiv  Detail & Related papers  (2026-05-21T08:25:17Z)
• SpecEyes: Accelerating Agentic Multimodal LLMs via Speculative Perception and Planning [104.01865949020304]
  Agentic multimodal large language models (MLLMs) achieve remarkable reasoning capabilities through iterative visual tool invocation.<n>However, the cascaded perception, reasoning, and tool-calling loops introduce significant sequential overhead.<n>This overhead, termed agentic depth, incurs prohibitive latency and seriously limits system-level throughput.<n>We propose SpecEyes, an agentic-level speculative acceleration framework that breaks this sequential bottleneck.
  arXiv  Detail & Related papers  (2026-03-24T17:45:47Z)
• Speculating Experts Accelerates Inference for Mixture-of-Experts [44.31811859704714]
  Mixture-of-Experts (MoE) models have gained popularity as a means of scaling the capacity of large language models (LLMs)<n>We propose an expert prefetching scheme that leverages currently computed internal model representations to speculate future experts.<n>Our approach achieves up to 14% reduction in time per output token over on-demand loading of experts from CPU memory.
  arXiv  Detail & Related papers  (2026-03-09T06:59:47Z)
• Timely Machine: Awareness of Time Makes Test-Time Scaling Agentic [72.97800570813175]
  We propose Timely Machine, redefining test-time as wall-clock time.<n>We introduce Timely-Eval, a benchmark spanning high-frequency tool calls, low-frequency tool calls, and time-constrained reasoning.<n>We find smaller models excel with fast feedback through more interactions, while larger models dominate high-latency settings via superior interaction quality.
  arXiv  Detail & Related papers  (2026-01-23T06:28:52Z)
• Task-oriented Learnable Diffusion Timesteps for Universal Few-shot Learning of Dense Tasks [48.86985692711283]
  Current diffusion model-based applications exploit the power of learned visual representations from multistep forward-backward Markovian processes for single-task prediction tasks.<n>We propose two modules: Task-aware Timestep Selection (TTS) to select ideal diffusion timesteps based on timestep-wise losses and similarity scores, and Timestep Feature Consolidation (TFC) to consolidate the selected timestep features.<n>Our framework effectively achieves superiority in dense prediction performance given only a few support queries.
  arXiv  Detail & Related papers  (2025-12-29T05:19:01Z)
• Reducing Latency of LLM Search Agent via Speculation-based Algorithm-System Co-Design [35.95362310928356]
  LLM-based search agents achieve strong performance but suffer from severe latency.<n>We revisit this bottleneck through the lens of speculation.<n>We present SPAgent, an algorithm-system co-design framework that expands the role of speculation in search agents to reduce latency.
  arXiv  Detail & Related papers  (2025-11-25T08:15:17Z)
• $\texttt{SPECS}$: Faster Test-Time Scaling through Speculative Drafts [55.231201692232894]
  $textttSPECS$ is a latency-aware test-time scaling method inspired by speculative decoding.<n>Our results show that $textttSPECS$matches or surpasses beam search accuracy while reducing latency by up to $sim$19.1%.
  arXiv  Detail & Related papers  (2025-06-15T05:50:05Z)
• SpecExtend: A Drop-in Enhancement for Speculative Decoding of Long Sequences [11.225649178057695]
  SpecExtend improves speculative decoding on long sequences without additional training.<n>To improve both draft accuracy and speed on long inputs without retraining, we propose Cross-model Retrieval.<n>SpecExtend accelerates speculative decoding by up to 2.84x on 16K-token long summarization and up to 3.86x on long reasoning.
  arXiv  Detail & Related papers  (2025-05-27T06:30:00Z)
• On Latency Predictors for Neural Architecture Search [8.564763702766776]
  We introduce a comprehensive suite of latency prediction tasks obtained in a principled way through automated partitioning of hardware device sets. We then design a general latency predictor to comprehensively study (1) the predictor architecture, (2) NN sample selection methods, (3) hardware device representations, and (4) NN operation encoding schemes. Building on conclusions from our study, we present an end-to-end latency predictor training strategy.
  arXiv  Detail & Related papers  (2024-03-04T19:59:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.