EvoSpec: Evolving Speculative Decoding via Real-Time Vocabulary and Parameter Adaptation

• URL: http://arxiv.org/abs/2605.27390v2
• Date: Thu, 28 May 2026 06:45:18 GMT
• Title: EvoSpec: Evolving Speculative Decoding via Real-Time Vocabulary and Parameter Adaptation
• Abstract summary: EvoSpec is a framework that enables real-time evolution of the draft model through dynamic vocabulary and parameter adaptation.<n>It achieves a 1.13x speedup in these settings over the state-of-the-art static baseline FR-Spec, with 27% lower memory overhead than standard online adaptation.
• Score: 15.94674618023941
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Speculative decoding accelerates Large Language Model inference via a draft-then-verify paradigm, yet the output projection layer becomes a bottleneck as vocabulary sizes scale. While existing static pruning methods effectively reduce this overhead, they suffer from precipitous drops in acceptance rate in specialized domains or topic-switching scenarios due to their inability to capture dynamic distribution shifts. To address this, we introduce EvoSpec, a framework that enables real-time evolution of the draft model through dynamic vocabulary and parameter adaptation. Unlike static or purely retrieval-based approaches, EvoSpec employs a context-aware mechanism that retrieves critical long-tail tokens via efficient semantic and statistical indexing. Furthermore, we propose a lightweight online alignment strategy utilizing curriculum learning to continually minimize the distributional gap between the draft and target models. Extensive evaluations across specialized domains (coding, law, and medicine) confirm that EvoSpec overcomes the limitations of static baselines. On EAGLE-3, it achieves a 1.13x speedup in these settings over the state-of-the-art static baseline FR-Spec, with 27\% lower memory overhead than standard online adaptation.

Related papers

• SlimSpec: Low-Rank Draft LM-Head for Accelerated Speculative Decoding [41.803148916760804]
  We propose SlimSpec, a low-rank parameterization of the drafter's LM-head that compresses the inner representation rather than the output.<n>SlimSpec achieves $4text-5times$ acceleration over the standard LM-head architecture.
  arXiv  Detail & Related papers  (2026-05-11T12:22:37Z)
• Dystruct: Dynamically Structured Diffusion Language Model Decoding via Bayesian Inference [51.12849550784653]
  Diffusion language models (DLMs) have emerged as a promising alternative to autoregressive models.<n>Most existing DLMs rely on a fixed generation length specified prior to decoding, which restricts their flexibility in real-world applications.<n>We propose a training-free, Bayesian structured decoding framework that formulates flexible-length generation as a dynamic structural inference problem.
  arXiv  Detail & Related papers  (2026-05-10T23:51:14Z)
• When Drafts Evolve: Speculative Decoding Meets Online Learning [15.718637402530467]
  Speculative decoding provides verification feedback that quantifies the deviation between the draft and target models at no additional cost.<n>We propose OnlineSpec, a unified framework that systematically leverages interactive feedback to continuously evolve draft models.<n>Our algorithms are equipped with theoretical justifications and improved acceleration rates, achieving up to 24% speedup over seven benchmarks and three foundation models.
  arXiv  Detail & Related papers  (2026-03-13T03:44:27Z)
• CAS-Spec: Cascade Adaptive Self-Speculative Decoding for On-the-Fly Lossless Inference Acceleration of LLMs [48.8252978488871]
  Speculative decoding offers seamless integration and broad utility when deploying large language models.<n>Cascading a hierarchy of draft models promises further acceleration and flexibility, but the high cost of training multiple models has limited its practical application.<n>We propose a novel Cascade Adaptive Self-Speculative Decoding (CAS-Spec) method which constructs speculative draft models.
  arXiv  Detail & Related papers  (2025-10-30T08:51:29Z)
• Lost in Translation? Vocabulary Alignment for Source-Free Adaptation in Open-Vocabulary Semantic Segmentation [90.5844979560448]
  VocAlign is a source-free domain adaptation framework specifically designed for VLMs in semantic segmentation.<n>Our approach achieves a notable 6.11 mIoU improvement on the CityScapes dataset and demonstrates superior performance on zero-shot segmentation benchmarks.
  arXiv  Detail & Related papers  (2025-09-18T17:59:58Z)
• HeteroSpec: Leveraging Contextual Heterogeneity for Efficient Speculative Decoding [2.2491281629568687]
  HeteroSpec is a speculative decoding framework that allocates verification effort in proportion to candidate uncertainty.<n>It delivers an average 4.24$times$ decoding speedup over state-of-the-art methods.<n>HeteroSpec requires no model retraining and remains compatible with other inference optimizations.
  arXiv  Detail & Related papers  (2025-05-19T15:38:40Z)
• Rethinking Addressing in Language Models via Contexualized Equivariant Positional Encoding [89.52931576290976]
  We present contextbfTextualized equivaritextbfAnt textbfPosition textbfEncoding (textbfTAPE), a novel framework that enhances positional embeddings by incorporating sequence content across layers.<n>Our method can be easily integrated into pre-trained transformers, offering parameter-efficient fine-tuning with minimal overhead.
  arXiv  Detail & Related papers  (2025-01-01T03:23:00Z)
• AdaEAGLE: Optimizing Speculative Decoding via Explicit Modeling of Adaptive Draft Structures [11.436315332919245]
  We introduce AdaEAGLE, the first SD framework that explicitly models adaptive draft structures.<n>AdaEAGLE achieves a $1.62times$ speedup over the vanilla AR decoding and outperforms fixed-length SotA baseline.
  arXiv  Detail & Related papers  (2024-12-25T13:57:33Z)
• Reference Trustable Decoding: A Training-Free Augmentation Paradigm for Large Language Models [79.41139393080736]
  Large language models (LLMs) have rapidly advanced and demonstrated impressive capabilities. In-Context Learning (ICL) and. Efficient Fine-Tuning (PEFT) are currently two mainstream methods for augmenting. LLMs to downstream tasks. We propose Reference Trustable Decoding (RTD), a paradigm that allows models to quickly adapt to new tasks without fine-tuning.
  arXiv  Detail & Related papers  (2024-09-30T10:48:20Z)
• Lightweight Modular Parameter-Efficient Tuning for Open-Vocabulary Object Detection [2.1155908599769764]
  We propose UniProj-Det, a lightweight modular framework for parameter-efficient open-vocabulary object detection.<n>UniProj-Det freezes pretrained backbones and introduces a Universal Projection module with a learnable modality token, enabling unified vision--language adaptation at minimal cost.
  arXiv  Detail & Related papers  (2024-08-20T12:27:53Z)
• Speculative Diffusion Decoding: Accelerating Language Generation through Diffusion [55.0194604505437]
  Speculative decoding has emerged as a widely adopted method to accelerate large language model inference.<n>This paper proposes an adaptation of speculative decoding which uses discrete diffusion models to generate draft sequences.
  arXiv  Detail & Related papers  (2024-08-10T21:24:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.