Delay-Adaptive Speculation Control for Low-Latency Edge-Cloud LLM Inference

• URL: http://arxiv.org/abs/2606.20591v1
• Date: Sun, 17 May 2026 07:15:30 GMT
• Title: Delay-Adaptive Speculation Control for Low-Latency Edge-Cloud LLM Inference
• Abstract summary: In distributed edge-clouds, draft length must be controlled online.<n>We show that the optimal draft length is a finite delay-monotone threshold.<n>We implement the method on a real edge-cloud testbed with a Jetson Orin Nano Super edge node and an3090 Ti cloud node.
• Score: 20.720083078911486
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Speculative decoding accelerates large language model (LLM) inference by using a lightweight draft model to propose tokens and a larger target model to verify them in parallel. In distributed edge-cloud inference, however, draft length must be controlled online: longer drafts amortize communication delay but reduce token acceptance, whereas shorter drafts preserve acceptance but trigger more communication rounds. We formulate this tradeoff as a ratio-type optimal stopping problem and prove that the optimal draft length is a finite delay-monotone threshold. The analysis identifies a critical delay below which single-token speculation is optimal and shows that the optimal length grows only logarithmically with communication delay. For time-varying networks, we extend the model to Markov-modulated channels and establish, under a bounded horizon and monotone stopping-region conditions, a state-dependent threshold policy. For unknown environments, we propose UCB-SpecStop, an online control algorithm with gap-free and gap-dependent expected regret bounds of $O(L_{\max}\sqrt{K_{\max}T\log(K_{\max}T)})$ and $O(\sum_{k:Δ_k>0}L_{\max}^2\log(K_{\max}T)/Δ_k)$. We implement the method on a real edge-cloud testbed with a Jetson Orin Nano Super edge node and an RTX~3090 Ti cloud node, using Qwen and Llama draft--target pairs. Experiments validate the predicted phase transition, with transition points near 83~ms and 111~ms. Qwen matches the geometric prediction, while Llama requires empirical-prefix calibration due to heavy-head acceptance. Across the tested delay grid, UCB-SpecStop reduces per-token latency over SpecDec++ by up to 22.4\%, approaches an offline oracle within 0.2--2.4\% in communication-dominated regimes, improves over naive UCB by up to 7.5\%, removes the 14.0--18.7\% gap caused by static tuning under delay drift, and gains 3.0--6.8\% with contextual channel-state information.

Related papers

• Strictly Causal Streaming Video Anomaly Detection with a Theoretically-Grounded State-Space Core [2.648500779572419]
  We introduce a causal streaming anomaly detector with no lookahead and no clip buffering.<n>Its temporal core is a diagonal linear state space recurrence with an input and state dependent decay gate.<n>We derive a closed form relationship between the recurrence decay spectrum and both detection delay and the shortest anomaly it can reliably capture.
  arXiv  Detail & Related papers  (2026-08-25T16:52:32Z)
• Constrained Decoding for Diffusion Language Models via Efficient Inference over Finite Automata [57.27430779838529]
  We present an exact and tractable algorithm for sampling from the constrained mean-field posterior under any constraint expressible as a finite automaton.<n>The approach guarantees constraint satisfaction by construction, supports both greedy and sampling-based decoding, and is compatible with parallel and block-wise decoding.<n> Empirical evaluations on Dream-7B and LLaDA-8 show substantial accuracy gains across various tasks.
  arXiv  Detail & Related papers  (2026-07-08T05:48:57Z)
• Capacity-Constrained Online Convex Optimization with Delayed Feedback [58.59385794080679]
  We study delayed online convex optimization (OCO) under a hard capacity constraint.<n>We find that for first-order feedback, capacity $C = (log T)$ suffices to recover standard delayed OCO rates up to logarithmic factors.<n>For bandit feedback, the regret rates are modulated by powers of $(1 + _textmax/C)$, where $_textmax$ is the maximum number of pending observations at any time.
  arXiv  Detail & Related papers  (2026-06-10T06:37:06Z)
• TAPS: Target-Aware Prefix Tree Selection for Diffusion-Drafted Speculative Decoding [4.5881678566958985]
  Using a diffusion model for parallel drafting is a promising approach for speculative decoding.<n>This shifts the bottleneck to verification: verifying a single sequence limits acceptance length, while verifying large draft trees incurs excessive target-model latency.<n>We propose TAPS, a target-aware prefix selection method that turns diffusion marginals into path-conditioned acceptance estimates.
  arXiv  Detail & Related papers  (2026-05-30T02:39:40Z)
• IGT-OMD: Implicit Gradient Transport for Decision-Focused Learning under Delayed Feedback [0.0]
  Decision-focused learning trains models end-to-end against downstream decision loss, but online settings suffer delayed feedback.<n>We identify emphstaleness amplification, a failure mode unique to bilevel optimization under delay.<n>We prove that any black-box delayed gradient incurs an irreducible regret cost from inner-solver approximation error.
  arXiv  Detail & Related papers  (2026-05-12T19:43:49Z)
• TARG: Training-Free Adaptive Retrieval Gating for Efficient RAG [46.122203287541005]
  Training-free Adaptive Retrieval Gating (TARG) is a single-shot policy that decides when to retrieve using only a short, no-latency draft from the base model.<n>On NQ-Open, TriviaQA, and PopQA, TARG consistently shifts the accuracy-efficiency frontier.
  arXiv  Detail & Related papers  (2025-11-12T23:09:52Z)
• Network-Optimised Spiking Neural Network (NOS) Scheduling for 6G O-RAN: Spectral Margin and Delay-Tail Control [7.382006859754025]
  This work presents a Network-Optimised Spiking (NOS) delay-aware scheduler for 6G radio access.<n>A small-signal analysis yields a delay-dependent threshold $k_star(Delta)$ and a spectral margin $delta = k_star(Delta) - gHrho(W)$ that compress topology, controller gain, and delay into a single design parameter.
  arXiv  Detail & Related papers  (2025-10-13T11:28:28Z)
• DiffuSpec: Unlocking Diffusion Language Models for Speculative Decoding [66.40658898418316]
  We present DiffuSpec, a training-free drop-in framework that uses a pretrained diffusion language model (DLM) to produce multi-token drafts in a single forward pass.<n>Across benchmarks, DiffuSpec yields up to 3x wall-clock speedup, establishing diffusion-based drafting as a robust alternative to autoregressive drafters for speculative decoding.
  arXiv  Detail & Related papers  (2025-09-28T07:00:15Z)
• Network-Optimised Spiking Neural Network for Event-Driven Networking [2.5941336499463383]
  Spiking neural networks offer event-driven computation suited to time-critical networking tasks such as anomaly detection, local routing control, and congestion management at the edge.<n>We introduce Network-Optimised Spiking (NOS), a compact two-variable unit whose state encodes normalised queue occupancy and a recovery resource.<n>We provide guidance for data-driven initialisation, surrogate-gradient training with a homotopy on reset sharpness, and explicit stability checks with topology of bounds for resource constrained deployments.
  arXiv  Detail & Related papers  (2025-09-27T22:31:24Z)
• Pipeline Parallelism is All You Need for Optimized Early-Exit Based Self-Speculative Decoding [73.67253077506672]
  Large language models (LLMs) deliver impressive generation quality, but incur very high inference cost.<n>Early-exit based self-speculative decoding (EESD) has emerged to mitigate this cost.<n>We propose Pipeline-Parallel Self-Speculative Decoding (PPSD) that fully pipelines the draft and verification work.
  arXiv  Detail & Related papers  (2025-09-19T04:51:41Z)
• Adaptive Deadline and Batch Layered Synchronized Federated Learning [66.93447103966439]
  Federated learning (FL) enables collaborative model training across distributed edge devices while preserving data privacy, and typically operates in a round-based synchronous manner.<n>We propose ADEL-FL, a novel framework that jointly optimize per-round deadlines and user-specific batch sizes for layer-wise aggregation.
  arXiv  Detail & Related papers  (2025-05-29T19:59:18Z)
• Stochastic Approximation with Delayed Updates: Finite-Time Rates under Markovian Sampling [73.5602474095954]
  We study the non-asymptotic performance of approximation schemes with delayed updates under Markovian sampling. Our theoretical findings shed light on the finite-time effects of delays for a broad class of algorithms.
  arXiv  Detail & Related papers  (2024-02-19T03:08:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.