Trees from Marginals: Autoregressive drafting with factorized priors

• URL: http://arxiv.org/abs/2607.06763v2
• Date: Sun, 12 Jul 2026 19:52:09 GMT
• Title: Trees from Marginals: Autoregressive drafting with factorized priors
• Abstract summary: Speculative decoding greatly increases the interactivity of autoregressive language models.<n>We introduce Weaver, a lightweight autoregressive adapter that constructs proposal trees from the top-K marginals of a factorized drafter.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Speculative decoding greatly increases the interactivity of autoregressive language models by trading off computation for extra tokens generated in a single forward pass. Factorized draft models are especially efficient because they predict future-token marginals in parallel, but their independence assumption causes acceptance rates to degrade sharply as the speculative budget grows. We analyze this limitation and introduce Weaver, a lightweight autoregressive adapter that constructs proposal trees from the top-K marginals of a factorized drafter. Weaver restores conditional dependencies between proposed tokens while avoiding a full-vocabulary projection. To support fast verification for models with Gated Delta Net layers, we derive a rollback-free tree-verification algorithm and implement optimized CUDA kernels in SGLang. By combining these model and systems contributions we achieve a 4.37-fold speedup over autoregressive decoding, and outperform a highly optimized DFlash baseline by 24.7%.

Related papers

• Bastion: Budget-Aware Speculative Decoding with Tree-structured Block Diffusion Drafting [36.83545196908392]
  BASTION is a budget-aware speculative decoding framework with tree-based diffusion drafting.<n>Our framework integrates three synergistic components: (1) an acceptance surrogate that estimates expected accepted length via path confidence, (2) an online latency estimator that calibrates a hardware-aware roofline model, and (3) an adaptive best-first expansion that grows the tree until marginal gains no longer justify incremental verification costs.
  arXiv  Detail & Related papers  (2026-05-28T10:21:34Z)
• SpecBound: Adaptive Bounded Self-Speculation with Layer-wise Confidence Calibration [13.454534256560558]
  Speculative decoding has emerged as a promising approach to accelerate autoregressive inference in large language models.<n>We propose a novel self-draft framework that suppresses spurious confidence via layer-wise temperature annealing in early-exit decision.<n>We show that our method achieves up to 2.33x wall-time speedup over standard autoregressive decoding.
  arXiv  Detail & Related papers  (2026-04-14T03:47:04Z)
• ECHO: Elastic Speculative Decoding with Sparse Gating for High-Concurrency Scenarios [12.103618967955846]
  Speculative Decoding promises to accelerate the inference of Large Language Models, yet its efficacy often degrades in production-grade serving.<n>We introduce ECHO, a high-oriented framework integrated into SGLang that reformulates speculative execution as a budgeted scheduling problem.<n>ECHO consistently outperforms SOTA methods in both low-load and high-load scenarios, achieving up to 5.35x walltime speedup and delivering over 20% relative speedup gain.
  arXiv  Detail & Related papers  (2026-03-10T03:51:24Z)
• Arbitrage: Efficient Reasoning via Advantage-Aware Speculation [71.45710345765528]
  Speculative Decoding accelerates inference by employing a fast but inaccurate draft model to autoregressively propose tokens.<n>But due to unnecessary rejections caused by token mismatches in semantically equivalent steps, traditional token-level Speculative Decoding struggles in reasoning tasks.<n>We propose Arbitrage, a novel step-level speculative generation framework that routes generation dynamically based on the relative advantage between draft and target models.
  arXiv  Detail & Related papers  (2025-12-04T17:50:53Z)
• Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios [76.85739138203014]
  We present SpecFormer, a novel architecture that accelerates unidirectional and attention mechanisms.<n>We demonstrate that SpecFormer achieves lower training demands and reduced computational costs.
  arXiv  Detail & Related papers  (2025-11-25T14:20:08Z)
• Fast Inference via Hierarchical Speculative Decoding [65.40448210801763]
  We introduce Hierarchical Speculative Decoding (HSD), an algorithm that stacks draft models into a hierarchy, where each model proposes tokens, and the next larger model verifies them in a single forward pass.<n>HSD gives up to 1.2x speed-up over the best single-draft baseline.
  arXiv  Detail & Related papers  (2025-10-22T15:56:19Z)
• R-Stitch: Dynamic Trajectory Stitching for Efficient Reasoning [80.104336426172]
  Chain-of-thought (CoT) enhances problem-solving ability of large language models.<n>CoT incurs substantial inference cost due to long autoregressive trajectories.<n>We introduce R-Stitch, a training-free hybrid decoding framework.
  arXiv  Detail & Related papers  (2025-07-23T08:14:36Z)
• Inference Acceleration of Autoregressive Normalizing Flows by Selective Jacobi Decoding [12.338918067455436]
  Normalizing flows are promising generative models with advantages such as theoretical rigor, analytical log-likelihood, and end-to-end training.<n>Recent advancements utilize autoregressive modeling, significantly enhancing expressive power and generation quality.<n>We propose a selective Jacobi decoding (SeJD) strategy that accelerates autoregressive inference through parallel iterative optimization.
  arXiv  Detail & Related papers  (2025-05-30T16:53:15Z)
• Latent Thought Models with Variational Bayes Inference-Time Computation [52.63299874322121]
  Latent Thought Models (LTMs) incorporate explicit latent thought vectors that follow an explicit prior model in latent space.<n>LTMs demonstrate superior sample and parameter efficiency compared to autoregressive models and discrete diffusion models.
  arXiv  Detail & Related papers  (2025-02-03T17:50:34Z)
• Graph-Structured Speculative Decoding [52.94367724136063]
  Speculative decoding has emerged as a promising technique to accelerate the inference of Large Language Models. We introduce an innovative approach utilizing a directed acyclic graph (DAG) to manage the drafted hypotheses. We observe a remarkable speedup of 1.73$times$ to 1.96$times$, significantly surpassing standard speculative decoding.
  arXiv  Detail & Related papers  (2024-07-23T06:21:24Z)
• Cascade Speculative Drafting for Even Faster LLM Inference [24.199907533534535]
  Speculative decoding improves the efficiency of large language model (LLM) inference.<n>We introduce Cascade Speculative Drafting, a speculative execution algorithm that incorporates two types of cascades.<n>CS Drafting achieves greater speedup compared to the baselines in our experiments, while preserving the same output distribution as the target model.
  arXiv  Detail & Related papers  (2023-12-18T18:59:46Z)
• Conditional Denoising Diffusion for Sequential Recommendation [62.127862728308045]
  Two prominent generative models, Generative Adversarial Networks (GANs) and Variational AutoEncoders (VAEs) GANs suffer from unstable optimization, while VAEs are prone to posterior collapse and over-smoothed generations. We present a conditional denoising diffusion model, which includes a sequence encoder, a cross-attentive denoising decoder, and a step-wise diffuser.
  arXiv  Detail & Related papers  (2023-04-22T15:32:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.