AdaptiveSD A Stability-Aware, Runtime-Adaptive Speculative Decoding Framework with Multi-Policy Orchestration for CPU-Constrained LLM Inference

• URL: http://arxiv.org/abs/2607.03876v1
• Date: Sat, 04 Jul 2026 13:39:40 GMT
• Title: AdaptiveSD A Stability-Aware, Runtime-Adaptive Speculative Decoding Framework with Multi-Policy Orchestration for CPU-Constrained LLM Inference
• Abstract summary: Fixed-depth speculative decoding has emerged as one promising technique, but in practice, it often leads to performance degradation due to bandwidth saturation, instability, or even catastrophic resource exhaustion resulting in system failure.<n>Our solution consists of four tightly-coupled components working together in a continuous feedback loop: an Adaptive Draft Controller, a Dynamic Policy Engine, a KV Cache Coordination Layer, and a Monitoring Engine.<n>While conventional approaches focus solely on maximizing throughput, we instead assess the effectiveness of our approach based on several key metrics including wasted drafted compute and inter-token latency dispersion alongside standard measures of speculative efficiency.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the rise of small quantized GGUF-based language models and their increasing use for on-device inference tasks, we have seen the growing need for an approach capable of reliably delivering these models at scale even under severe memory bandwidth constraints such as those imposed by pure CPU implementations. Fixed-depth speculative decoding has emerged as one promising technique, but in practice, it often leads to performance degradation due to either bandwidth saturation, instability, or even catastrophic resource exhaustion resulting in system failure. To overcome this problem, we introduce AdaptiveSD, a fully runtime-adaptive speculative decoding framework aimed at ensuring robust, reliable execution across the spectrum of model types and workloads. Our solution consists of four tightly-coupled components working together in a continuous feedback loop: a Runtime Monitoring Engine tracking multiple signals relevant to ongoing computation, an Adaptive Draft Controller enforcing an eleven rule policy hierarchy prioritizing system resource preservation over raw draft count, a Dynamic Policy Engine employing a suite of heuristic and reinforcement learning techniques to dynamically modify policies depending upon workload behavior, and finally, a KV Cache Coordination Layer managing cache states with fine-grained control through INT8 shadow buffers and position aware evictions. While conventional approaches focus solely on maximizing throughput, we instead assess the effectiveness of our approach based on several key metrics including wasted drafted compute and inter-token latency dispersion alongside standard measures of speculative efficiency.

Related papers

• Harmonizing Real-Time Constraints and Long-Horizon Reasoning: An Asynchronous Agentic Framework for Dynamic Scheduling [11.212756680882444]
  We introduce RACE-Sched, an agent-based framework that decouples policy execution from logical reasoning via a dual-stream architecture.<n>This approach harmonizes real-time constraints with long-horizon reasoning to achieve superior solution quality and robust adaptation to dynamic events.
  arXiv  Detail & Related papers  (2026-05-28T02:26:18Z)
• Safe Multi-Agent Deep Reinforcement Learning for Privacy-Aware Edge-Device Collaborative DNN Inference [8.14391361533752]
  This paper proposes a privacy-aware collaborative inference framework, in which adaptive model partitioning is performed across edge devices and servers.<n>We formulate the joint problem as a Constrained Markov Decision Process (CMDP) that integrates model deployment, user-server association, model partitioning, and resource allocation.<n>We show that HC-MAPPO-L consistently satisfies stringent delay constraints while achieving a superior balance among energy consumption and privacy cost.
  arXiv  Detail & Related papers  (2026-02-23T11:33:52Z)
• TS-DP: Reinforcement Speculative Decoding For Temporal Adaptive Diffusion Policy Acceleration [64.32072516882947]
  Diffusion Policy excels in embodied control but suffers from high inference latency and computational cost.<n>We propose Temporal-aware Reinforcement-based Speculative Diffusion Policy (TS-DP)<n>TS-DP achieves up to 4.17 times faster inference with over 94% accepted drafts, reaching an inference frequency of 25 Hz.
  arXiv  Detail & Related papers  (2025-12-13T07:53:14Z)
• Training-free Context-adaptive Attention for Efficient Long Context Modeling [57.703159205740185]
  Training-free Context-adaptive Attention (TCA-Attention) is a training-free sparse attention mechanism that selectively attends to only the informative tokens for efficient long-context inference.<n>TCA-Attention achieves a 2.8$times$ speedup and reduces KV cache by 61% at 128K context length while maintaining performance comparable to full attention.
  arXiv  Detail & Related papers  (2025-12-10T01:54:57Z)
• Benchmarking Dynamic SLO Compliance in Distributed Computing Continuum Systems [9.820223170841219]
  Service Level Objectives (SLOs) in large-scale architectures are challenging due to their heterogeneous nature and varying service requirements.<n>We present a benchmark of Active Inference -- an emerging method from neuroscience -- against three established reinforcement learning algorithms.<n>We find that Active Inference is a promising approach for ensuring SLO compliance in DCCS, offering lower memory usage, stable CPU utilization, and fast convergence.
  arXiv  Detail & Related papers  (2025-03-05T08:56:26Z)
• Bisimulation metric for Model Predictive Control [44.301098448479195]
  Bisimulation Metric for Model Predictive Control (BS-MPC) is a novel approach that incorporates bisimulation metric loss in its objective function to directly optimize the encoder. BS-MPC improves training stability, robustness against input noise, and computational efficiency by reducing training time. We evaluate BS-MPC on both continuous control and image-based tasks from the DeepMind Control Suite.
  arXiv  Detail & Related papers  (2024-10-06T17:12:10Z)
• Digital Twin-Assisted Data-Driven Optimization for Reliable Edge Caching in Wireless Networks [60.54852710216738]
  We introduce a novel digital twin-assisted optimization framework, called D-REC, to ensure reliable caching in nextG wireless networks. By incorporating reliability modules into a constrained decision process, D-REC can adaptively adjust actions, rewards, and states to comply with advantageous constraints.
  arXiv  Detail & Related papers  (2024-06-29T02:40:28Z)
• Action-Quantized Offline Reinforcement Learning for Robotic Skill Learning [68.16998247593209]
  offline reinforcement learning (RL) paradigm provides recipe to convert static behavior datasets into policies that can perform better than the policy that collected the data. In this paper, we propose an adaptive scheme for action quantization. We show that several state-of-the-art offline RL methods such as IQL, CQL, and BRAC improve in performance on benchmarks when combined with our proposed discretization scheme.
  arXiv  Detail & Related papers  (2023-10-18T06:07:10Z)
• Non-stationary Online Learning with Memory and Non-stochastic Control [71.14503310914799]
  We study the problem of Online Convex Optimization (OCO) with memory, which allows loss functions to depend on past decisions. In this paper, we introduce dynamic policy regret as the performance measure to design algorithms robust to non-stationary environments. We propose a novel algorithm for OCO with memory that provably enjoys an optimal dynamic policy regret in terms of time horizon, non-stationarity measure, and memory length.
  arXiv  Detail & Related papers  (2021-02-07T09:45:15Z)
• Enforcing robust control guarantees within neural network policies [76.00287474159973]
  We propose a generic nonlinear control policy class, parameterized by neural networks, that enforces the same provable robustness criteria as robust control. We demonstrate the power of this approach on several domains, improving in average-case performance over existing robust control methods and in worst-case stability over (non-robust) deep RL methods.
  arXiv  Detail & Related papers  (2020-11-16T17:14:59Z)
• Trajectory Optimization for Nonlinear Multi-Agent Systems using Decentralized Learning Model Predictive Control [5.2647625557619815]
  We present a decentralized minimum-time trajectory optimization scheme based on learning model predictive control for multi-agent systems with nonlinear decoupled dynamics and coupled state constraints. Our framework results in a decentralized controller, which requires no communication between agents over each iteration of task execution, and guarantees persistent feasibility, finite-time closed-loop convergence, and non-decreasing performance of the global system over task iterations.
  arXiv  Detail & Related papers  (2020-04-02T23:04:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.