AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding

• URL: http://arxiv.org/abs/2607.25852v2
• Date: Wed, 29 Jul 2026 07:31:33 GMT
• Title: AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding
• Abstract summary: Speculative decoding accelerates large language model inference without changing the target distribution.<n>We present AngelSpec, a unified training framework for speculative decoding.<n>At the architecture level, we propose DFly, a block-diffusion framework.
• Score: 29.797292873854
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Speculative decoding accelerates large language model inference without changing the target distribution, but no single drafting structure performs best across real-world workloads. Autoregressive multi-token prediction (MTP) is a lightweight, stable proposal mechanism, whereas block-parallel diffusion amortizes drafting latency over much longer candidate sequences; the better choice depends strongly on the output distribution. We present AngelSpec, a unified training framework for MTP and block-parallel speculative decoding that addresses this heterogeneity at three levels. At the training level, rather than fitting one universal drafter to a uniform data mixture, we co-specialize structure and data: the MTP drafter is trained on diverse conversational data for high-entropy open-ended chat, and the block-diffusion drafter on code and mathematics data for longer predictable continuations. At the architecture level, we propose DFly, a block-diffusion framework combining a hybrid target-conditioning backbone with a predecessor-conditioned autoregressive head, improving target-feature utilization and intra-block dependency modeling while keeping generation parallel. At the inference level, both acceptance length and verification cost vary with domain, request, online load, and hardware, so DFly treats verification as a shared batch-level resource: it reallocates compute toward high-confidence prefixes across requests and combines expected utility with a profiled cost model to adapt verification depth online. Across the Hy3 series, DFly raises the average accepted length on Hy3-A21B by roughly 30% and attains the highest average throughput at every tested concurrency from 4 to 64, a 1.98-2.40x speedup over autoregressive decoding and 10.5-11.8% higher throughput than DFlash. We release AngelSpec to support training and extending these methods.

Related papers

• Faster Than Flash: Exploiting Attention Sparsity for Efficient Long-Context Decoding [51.48425605758328]
  We present Faster Flash Decoding (FFD), a novel hardware-algorithm co-design framework designed to break the memory wall in long-context decoding.<n>FFD integrates the selector and computer into a fully fused kernel, replacing external metadata indices with content-aware scanning.<n>It achieves up to 11.6x kernel-level speedup and scaling to 256K context length, with 2.37x end-to-end throughput improvement.
  arXiv  Detail & Related papers  (2026-08-31T15:13:20Z)
• DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation [69.8840101036735]
  Speculative decoding accelerates Large Language Model (LLM) inference.<n>Recent parallel drafters efficiently propose long token sequences in a single forward pass, but suffer from rapid acceptance decay due to a lack of inter-token dependencies.<n>We introduce DSpark, a speculative decoding framework that unifies high- throughput parallel generation with adaptive, load-aware verification.
  arXiv  Detail & Related papers  (2026-07-06T14:28:06Z)
• HyperDFlash: Hyper-Connection-Aligned Block Speculative Decoding with Gated Residual Reduction [8.146414118644069]
  We present HyperDFlash, a block-parallel speculative decoding framework tailored to DeepSeek-V4's Hyper-Connections (HC)<n>Despite the strong performance of DeepSeek-V4's native Multi-Token Prediction (MTP) module on initial token drafting, its draft accuracy degrades sharply at later positions.<n>Although the original DFlash method supports efficient one-pass block drafting, it cannot be seamlessly adapted to the HC paradigm.
  arXiv  Detail & Related papers  (2026-06-25T08:31:53Z)
• How Far Can Disaggregation Go? A Design-Space Exploration of Attention-FFN Disaggregation for Efficient MoE LLM Serving [14.552785121277529]
  We study the benefits and limits of Attention-FFN Disaggregation (AFD) at scale through a framework that fuses on-device kernel measurements with high-fidelity network simulation.<n>Under strict TTFT/TPOT SLOs, AFD sustains around 4k tokens/s of system throughput on DeepSeek-V3.2 across chat, coding, and agentic-coding workloads.
  arXiv  Detail & Related papers  (2026-05-27T10:55:57Z)
• SlimSpec: Low-Rank Draft LM-Head for Accelerated Speculative Decoding [41.803148916760804]
  We propose SlimSpec, a low-rank parameterization of the drafter's LM-head that compresses the inner representation rather than the output.<n>SlimSpec achieves $4text-5times$ acceleration over the standard LM-head architecture.
  arXiv  Detail & Related papers  (2026-05-11T12:22:37Z)
• DiffuSpec: Unlocking Diffusion Language Models for Speculative Decoding [66.40658898418316]
  We present DiffuSpec, a training-free drop-in framework that uses a pretrained diffusion language model (DLM) to produce multi-token drafts in a single forward pass.<n>Across benchmarks, DiffuSpec yields up to 3x wall-clock speedup, establishing diffusion-based drafting as a robust alternative to autoregressive drafters for speculative decoding.
  arXiv  Detail & Related papers  (2025-09-28T07:00:15Z)
• Accelerating Diffusion LLMs via Adaptive Parallel Decoding [60.407727995313074]
  We introduce adaptive parallel decoding (APD), a novel method that dynamically adjusts the number of tokens sampled in parallel.<n>APD provides markedly higher throughput with minimal quality degradations on downstream benchmarks.
  arXiv  Detail & Related papers  (2025-05-31T06:10:10Z)
• SpecRouter: Adaptive Routing for Multi-Level Speculative Decoding in Large Language Models [21.933379266533098]
  Large Language Models (LLMs) present a critical trade-off between inference quality and computational cost.<n>Existing serving strategies often employ fixed model scales or static two-stage speculative decoding.<n>This paper introduces systemname, a novel framework that reimagines LLM inference as an adaptive routing problem.
  arXiv  Detail & Related papers  (2025-05-12T15:46:28Z)
• Jakiro: Boosting Speculative Decoding with Decoupled Multi-Head via MoE [15.003006630308517]
  Speculative decoding (SD) accelerates large language model inference by using a smaller draft model to predict multiple tokens.<n>We propose Jakiro, leveraging Mixture of Experts (MoE), where independent experts generate diverse predictions.<n>Our method significantly boosts prediction accuracy and achieves higher inference speedups.
  arXiv  Detail & Related papers  (2025-02-10T09:24:06Z)
• ALOcc: Adaptive Lifting-Based 3D Semantic Occupancy and Cost Volume-Based Flow Predictions [91.55655961014027]
  3D semantic occupancy and flow prediction are fundamental to understanding scene scene.<n>This paper proposes a vision-based framework with three targeted improvements.<n>Our purely convolutional architecture establishes new SOTA performance on multiple benchmarks for both semantic occupancy and joint semantic-flow prediction.
  arXiv  Detail & Related papers  (2024-11-12T11:32:56Z)
• Fed-CVLC: Compressing Federated Learning Communications with Variable-Length Codes [54.18186259484828]
  In Federated Learning (FL) paradigm, a parameter server (PS) concurrently communicates with distributed participating clients for model collection, update aggregation, and model distribution over multiple rounds. We show strong evidences that variable-length is beneficial for compression in FL. We present Fed-CVLC (Federated Learning Compression with Variable-Length Codes), which fine-tunes the code length in response to the dynamics of model updates.
  arXiv  Detail & Related papers  (2024-02-06T07:25:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.