When Hidden States Drift: Can KV Caches Rescue Long-Range Speculative Decoding?

• URL: http://arxiv.org/abs/2604.26412v1
• Date: Wed, 29 Apr 2026 08:25:01 GMT
• Title: When Hidden States Drift: Can KV Caches Rescue Long-Range Speculative Decoding?
• Abstract summary: Speculative decoding accelerates LLM inference, but SOTA hidden-state-based drafters suffer from long-range decay.<n>We revisit long-range decay from the perspective of context information preservation.<n>We introduce KVShot, a diagnostic framework that compares three reuse paradigms.
• Score: 19.77569541429818
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Speculative decoding accelerates LLM inference, but SOTA hidden-state-based drafters suffer from long-range decay: draft accuracy degrades as the speculative step increases. Existing work attributes this decay to train-inference mismatch and proposes test-time training (TTT) as a remedy, yet we observe that long-range decay persists even in TTT-trained drafters. We revisit long-range decay from the perspective of context information preservation. In hidden-state reuse, we argue the target hidden state acts as a biased context compression: it aggregates historical token information according to the attention query at the current position, yielding a compact representation optimized for immediate next-token prediction. This compression can suppress information less relevant to the current query but important for later speculative steps. In contrast, the target model's KV cache serves as an explicit context, retaining the complete set of token-wise KV representations. We therefore posit the KV-Reuse Hypothesis: allowing the draft model to reuse the target KV cache can provide richer signals for long-horizon drafting. To test this hypothesis, we introduce KVShot, a diagnostic framework that compares three reuse paradigms: hidden-only, KV-only, and hybrid. Extensive evaluations on Qwen3-8B show that KV-Reuse improves long-range acceptance, although end-to-end speedups remain marginal under current training pipelines. Our analysis identifies two key structural bottlenecks: shallow drafters struggle to estimate target queries accurately, and draft-side KV projections receive sparse gradient signals. These findings suggest that realizing the full potential of KV-aware decoding requires moving beyond TTT toward block-wise training paradigms. By exposing these bottlenecks, KVShot provides a foundational diagnostic testbed and a clear roadmap for designing next-generation inference architectures.

Related papers

• BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference [20.58039191111583]
  Large Reasoning Models (LRMs) achieve superior problem-solving through extended Chain-of-Thought (CoT) generation.<n>The resulting key-value ( KV) cache grows linearly with sequence length and creates severe memory bottlenecks.<n>We propose BeaconKV, a training-free KV cache compression method that maintains beacon queries, compact representatives for each global query cluster.
  arXiv  Detail & Related papers  (2026-09-04T10:23:16Z)
• CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration [67.17709877327232]
  Cate KV is a hybrid KV cache method that retains only critical token information for consistent heads.<n>Cate KV reduces memory usage by up to $2.72times$ and accelerates decoding by $2.18times$ in single-sample inputs.
  arXiv  Detail & Related papers  (2026-08-31T06:02:37Z)
• A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding [11.263048239748379]
  We introduce SparseSpec-L, a training-free self-speculative decoding framework for long-context inference.<n>SparseSpec-L generates lightweight drafts directly from the target model using a dynamically sparsified and recallable KV cache.<n> Experiments across multiple long-context tasks and model scales show consistent end-to-end acceleration.
  arXiv  Detail & Related papers  (2026-07-30T06:16:56Z)
• Information-Aware KV Cache Compression for Long Reasoning [71.77807044404499]
  Existing KV cache compression methods mainly rely on attention weights to estimate token importance.<n>We propose textbfInfoKV, an entropy-aware KV cache compression framework that incorporates information-theoretic signals.
  arXiv  Detail & Related papers  (2026-06-25T11:03:37Z)
• ArborKV: Structure-Aware KV Cache Management for Scaling Tree-based LLM Reasoning [8.78883720506938]
  Tree-of-Thoughts (ToT) organizes inference to tree-structured search with branching and backtracking.<n> retaining KV states for a frontier of partial trajectories quickly becomes a memory bottleneck.<n>We propose ArborKV, a structure-aware eviction framework that couples a lightweight value estimator with a tree-aware allocation policy.
  arXiv  Detail & Related papers  (2026-05-21T07:40:57Z)
• Predicting Future Utility: Global Combinatorial Optimization for Task-Agnostic KV Cache Eviction [19.14455067106419]
  Current KV cache eviction methods rely on instantaneous metrics, implicitly assuming that score magnitudes are consistent proxies for importance across all heads.<n>In this paper, we propose that optimal budget allocation should be governed by the marginal utility in preserving long-term semantic information.<n>We implement a data-driven offline profiling protocol to facilitate the practical deployment of LU-KV.
  arXiv  Detail & Related papers  (2026-02-09T12:23:38Z)
• ForesightKV: Optimizing KV Cache Eviction for Reasoning Models by Learning Long-Term Contribution [84.41751286055909]
  We develop a training-based KV cache eviction framework that learns to predict which KV pairs to evict during longtext generations.<n>We formulate cache eviction as a Markov Decision Process and apply the GRPO algorithm to mitigate the significant language modeling loss increase on low-entropy tokens.
  arXiv  Detail & Related papers  (2026-02-03T07:16:51Z)
• FAEDKV: Infinite-Window Fourier Transform for Unbiased KV Cache Compression [18.12657364501536]
  FAEDKV is a novel, training-free KV cache compression framework.<n>It preserves both early and recent contextual information.<n>Experiments on LongBench benchmark demonstrate FAEDKV's superiority over existing methods by up to 22%.
  arXiv  Detail & Related papers  (2025-07-26T18:20:25Z)
• KV-Latent: Dimensional-level KV Cache Reduction with Frequency-aware Rotary Positional Embedding [72.12756830560217]
  Large language models (LLMs) based on Transformer Decoders have become the preferred choice for conversational generative AI.<n>Despite the overall superiority of the Decoder architecture, the gradually increasing Key-Value cache during inference has emerged as a primary efficiency bottleneck.<n>By down-sampling the Key-Value vector dimensions into a latent space, we can significantly reduce the KV Cache footprint and improve inference speed.
  arXiv  Detail & Related papers  (2025-07-15T12:52:12Z)
• Draft-based Approximate Inference for LLMs [7.287280338330983]
  We propose a novel framework for approximate Large Language Models inference.<n>We introduce two instantiations of our proposed framework: (i) SpecKV, the first method that leverages a draft output to accurately assess the importance of each KV pair for more effective KV cache dropping, and (ii) SpecPC, which uses the draft model's attention activations to identify and discard unimportant prompt tokens.<n>Our methods consistently achieve higher accuracy than existing baselines, while preserving the same improvements in memory usage, latency, and throughput.
  arXiv  Detail & Related papers  (2025-06-10T02:37:46Z)
• Activation-aware Probe-Query: Effective Key-Value Retrieval for Long-Context LLMs Inference [56.71209737306054]
  We propose textbfActQKV, a training-free, textbfActivation-aware approach that dynamically determines probe-textbfQuery and leverages it to retrieve the relevant textbfKV pairs for inference.<n>Experiments on the Long-Bench and $infty$ Benchmarks demonstrate its state-of-the-art performance with competitive inference quality and resource efficiency.
  arXiv  Detail & Related papers  (2025-02-19T08:50:44Z)
• More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compression [71.42818367729573]
  In large language models (LLMs), the memory usage of KV cache has become a critical bottleneck during inference.<n>The mainstream KV compression methods, including KV pruning and KV quantization, primarily focus on either token or precision dimension separately.<n>In this paper, we comprehensively investigate the token-precision trade-off in KV cache compression.
  arXiv  Detail & Related papers  (2024-12-17T09:20:31Z)
• PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generation [97.41972925670508]
  Large vision-language models (LVLMs) incur significant computational and memory overhead during inference.<n>We present PrefixKV, where "Prefix" means the top-ranked KV based on importance rather than position in the original sequence.<n>Our method achieves the state-of-the-art performance compared with others.
  arXiv  Detail & Related papers  (2024-12-04T15:48:59Z)
• No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization [31.806112535762367]
  Key-Value (KV) Caching has become an essential technique for accelerating the inference speed and throughput of generative Large Language Models(LLMs)
  arXiv  Detail & Related papers  (2024-02-28T06:34:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.