SPORK: Self-Speculative Forking to Accelerate Agentic LLM Inference

• URL: http://arxiv.org/abs/2607.03333v1
• Date: Fri, 03 Jul 2026 13:51:32 GMT
• Title: SPORK: Self-Speculative Forking to Accelerate Agentic LLM Inference
• Abstract summary: SPORK dispatches the speculated tool call early, overlapping its execution with the remaining chain-of-thought decode.<n>SPORK is a training-free controller that dispatches the speculated tool call early, overlapping its execution with the remaining chain-of-thought decode.
• Score: 6.978795233982147
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: LLM agents are becoming a common interface for research, coding, and question answering, yet their Thought-Action-Observation loop is often serial: the model reasons, emits a tool call, then idles the GPU until the result returns. This wait consumes 16-37% of wall time in our workloads and 35-61% in prior reports. Speculative tool execution can hide this wait, but existing systems need auxiliary predictors, historical traces, or static workflow graphs, leaving a gap for training-free, day-one deployment. We observe that the model can be its own predictor: a probe forked at the start of generation predicts Qwen3-32B's upcoming tool name with 74.6-99.6% accuracy across five benchmarks. We present SPORK (Self-sPeculative fORKing), a training-free controller that dispatches the speculated tool call early, overlapping its execution with the remaining chain-of-thought decode. A cost model captures when speculation breaks even, and each component improves one of its terms: a prefix-cache fork cuts probe cost, a confidence gate filters mispredictions, and partial-token accept turns rejected probes into speculative-decoding drafts. On acceptance, the tool result is ready when reasoning ends; on rejection, SPORK falls back to serial execution with no correctness penalty. On real-tool benchmarks, SPORK cuts Qwen3-32B's GAIA P95 by 18% (131.9 to 108.1 s); the mechanism holds across model sizes from 4B to 32B and across dense and mixture-of-experts models, with task accuracy within 1 pp of baseline or better wherever measured. SPORK deploys as a thin controller over standard completion APIs (no retraining, no auxiliary models, no offline traces) and is orthogonal to token-level speculative decoding. SPORK is open source at https://github.com/baihuajun24/spork.

Related papers

• How to Speculate about Uncertainty in Agentic Coding? A Draft-Model Gate Method [4.37919186746582]
  We present Speculative Uncertainty (SU), a method that recovers a predictive failure signal for a black-box agent.<n>We extract phase-aware features by separating the reasoning and action spans, and calibrate them against a verifiable objective.<n>SU produces a failure-likelihood score that any downstream policy, such as routing, human intervention, or extra test-time compute, can consume directly.
  arXiv  Detail & Related papers  (2026-09-04T15:30:22Z)
• Large Models for Small Devices: Recent Advances and Empirical Analysis of Edge AI Deployment [4.582004238093247]
  Running large AI models on resource-constrained edge devices requires model compression to reduce model size and computation.<n>We deploy compact language and image models on GPU, CPU, and Raspberry Pi platforms across question answering and image segmentation.<n>We explain these effects through neural-flow graph analysis and prefill-decode-level latency decomposition.
  arXiv  Detail & Related papers  (2026-08-16T11:52:13Z)
• AgentDebugX: An Open-Source Toolkit for Failure Observability, Attribution, and Recovery in LLM Agents [83.37969790806069]
  LLM agent failures are difficult to debug because the step where an error surfaces is often not the one that caused it.<n>We present Agent DebuggerX, an open-source framework that organizes debug as a closed loop of Detect, Attribute, Recover, and Rerun.<n>At its core, Deep Debugger performs multi-turn root-cause diagnosis through global trajectory understanding.
  arXiv  Detail & Related papers  (2026-07-21T06:21:13Z)
• Calibrated e-CUSUM Decoding for Quantized Reasoning Models: Why Token Log-Probability Is the Wrong Observable for Decoding Monitors [0.6999740786886536]
  Low-bit quantization makes small reasoning models inexpensive to deploy but can degrade their chains of thought.<n>We show that the centered token log-probability increment $log p(w_t)+H_t$ is the wrong observable for this purpose.<n>We introduce a training-free decoding controller that combines a degeneration-aware alarm score fusing token uncertainty with explicit repetition.
  arXiv  Detail & Related papers  (2026-07-13T09:34:36Z)
• When Does Streaming Tool Use Help? Characterizing Tool-Intent Stabilization in Streaming Retrieval-Augmented Generation [0.0]
  Streaming Retrieval-Augmented Generation hides tool latency by issuing retrieval queries in parallel with the user's still-arriving input, before the utterance is complete.<n>We name and measure this property, tool-intent stabilization: the point in the input stream at which a speculative query's retrieval converges on the answer-bearing result.
  arXiv  Detail & Related papers  (2026-06-18T11:38:17Z)
• Auditable Graph-Guided Root Cause Analysis for Kubernetes Incidents [1.116726665785374]
  We present Graph Traversal Agent, a graph-guided RCA agent that combines LLM reasoning with specialized tools.<n>We map operational constraints, including read-only evidence collection, propagation-aware diagnosis, bounded execution, and independently validated verdicts.<n>On ITBench snapshots scored by one fixed qwen-plus judge, the audited system raises root-cause-entity F1 over an earlier iteration of the same system.
  arXiv  Detail & Related papers  (2026-06-07T12:05:09Z)
• LLM Agents Already Know When to Call Tools -- Even Without Reasoning [25.40369702634587]
  LLM agents tend to call tools indiscriminately, even when the model can answer directly.<n>No existing benchmark systematically studies when a tool call is actually needed.<n>We propose When2Tool, a benchmark of 18 environments spanning three categories of tool necessity.
  arXiv  Detail & Related papers  (2026-05-10T01:37:40Z)
• Can We Predict Before Executing Machine Learning Agents? [74.39460101251792]
  We formalize the task of Data-centric Solution Preference and construct a comprehensive corpus of 18,438 pairwise comparisons.<n>We demonstrate that LLMs exhibit significant predictive capabilities when primed with a Verified Data Analysis Report.<n>We instantiate this framework in FOREAGENT, an agent that employs a Predict-then-Verify loop, achieving a 6x acceleration in convergence while surpassing execution-based baselines by +6%.
  arXiv  Detail & Related papers  (2026-01-09T16:44:17Z)
• LYNX: Learning Dynamic Exits for Confidence-Controlled Reasoning [15.597220136913258]
  LYNX is an online early-exit mechanism that turns a model's own hidden-state awareness into confidence-controlled stopping decisions.<n>We train and calibrate this probe once on a generic mathematical corpus and reuse it unchanged across benchmarks, decoding temperatures, and even non-mathematical tasks.
  arXiv  Detail & Related papers  (2025-12-05T00:04:42Z)
• $\texttt{SPECS}$: Faster Test-Time Scaling through Speculative Drafts [55.231201692232894]
  $textttSPECS$ is a latency-aware test-time scaling method inspired by speculative decoding.<n>Our results show that $textttSPECS$matches or surpasses beam search accuracy while reducing latency by up to $sim$19.1%.
  arXiv  Detail & Related papers  (2025-06-15T05:50:05Z)
• MICE for CATs: Model-Internal Confidence Estimation for Calibrating Agents with Tools [54.63478102768333]
  Well-calibrated model confidences can be used to weigh the risk versus reward of potential actions.<n>We propose a novel class of model-internal confidence estimators (MICE) to better assess confidence when calling tools.
  arXiv  Detail & Related papers  (2025-04-28T18:06:38Z)
• Enhancing Self-Consistency and Performance of Pre-Trained Language Models through Natural Language Inference [72.61732440246954]
  Large pre-trained language models often lack logical consistency across test inputs. We propose a framework, ConCoRD, for boosting the consistency and accuracy of pre-trained NLP models. We show that ConCoRD consistently boosts accuracy and consistency of off-the-shelf closed-book QA and VQA models.
  arXiv  Detail & Related papers  (2022-11-21T21:58:30Z)
• A shallow neural model for relation prediction [2.2559617939136505]
  We propose a shallow neural model (SHALLOM) that accurately infers missing relations from entities. Our experiments indicate that SHALLOM outperforms state-of-the-art approaches on the FB15K-237 and WN18RR datasets.
  arXiv  Detail & Related papers  (2021-01-22T13:10:11Z)
• Probing Model Signal-Awareness via Prediction-Preserving Input Minimization [67.62847721118142]
  We evaluate models' ability to capture the correct vulnerability signals to produce their predictions. We measure the signal awareness of models using a new metric we propose- Signal-aware Recall (SAR) The results show a sharp drop in the model's Recall from the high 90s to sub-60s with the new metric.
  arXiv  Detail & Related papers  (2020-11-25T20:05:23Z)
• Superiority of Simplicity: A Lightweight Model for Network Device Workload Prediction [58.98112070128482]
  We propose a lightweight solution for series prediction based on historic observations. It consists of a heterogeneous ensemble method composed of two models - a neural network and a mean predictor. It achieves an overall $R2$ score of 0.10 on the available FedCSIS 2020 challenge dataset.
  arXiv  Detail & Related papers  (2020-07-07T15:44:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.