SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding

• URL: http://arxiv.org/abs/2604.09557v1
• Date: Tue, 10 Feb 2026 16:19:56 GMT
• Title: SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding
• Abstract summary: Speculative Decoding (SD) has emerged as a critical technique for accelerating Large Language Model (LLM) inference.<n>Previous benchmarks suffer from limited task diversity, inadequate support for throughput-oriented evaluation, and a reliance on high-level implementations that fail to reflect production environments.<n>We introduce SPEED-Bench, a comprehensive suite designed to standardize SD evaluation across diverse semantic domains and realistic serving regimes.
• Score: 3.876913658180685
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Speculative Decoding (SD) has emerged as a critical technique for accelerating Large Language Model (LLM) inference. Unlike deterministic system optimizations, SD performance is inherently data-dependent, meaning that diverse and representative workloads are essential for accurately measuring its effectiveness. Existing benchmarks suffer from limited task diversity, inadequate support for throughput-oriented evaluation, and a reliance on high-level implementations that fail to reflect production environments. To address this, we introduce SPEED-Bench, a comprehensive suite designed to standardize SD evaluation across diverse semantic domains and realistic serving regimes. SPEED-Bench offers a carefully curated Qualitative data split, selected by prioritizing semantic diversity across the data samples. Additionally, it includes a Throughput data split, allowing speedup evaluation across a range of concurrencies, from latency-sensitive low-batch settings to throughput-oriented high-load scenarios. By integrating with production engines like vLLM and TensorRT-LLM, SPEED-Bench allows practitioners to analyze system behaviors often masked by other benchmarks. We highlight this by quantifying how synthetic inputs overestimate real-world throughput, identifying batch-size dependent optimal draft lengths and biases in low-diversity data, and analyzing the caveats of vocabulary pruning in state-of-the-art drafters. We release SPEED-Bench to establish a unified evaluation standard for practical comparisons of SD algorithms.

Related papers

• DICS: Exploring Data Intrinsic Consistency for Visual Instruction Selection [53.457992197673185]
  Data Intrinsic Consistency (DIC) is a self-scoring metric designed to quantify the sample-level inter-component consistency.<n>DICS is an adaptive data selection method that optimize the trade-off between high intra-sample consistency and global distributional diversity.<n>DIC-6M is a 6M-sample multi-modal instruction corpus that enables the largest-scale visual instruction selection study to date.
  arXiv  Detail & Related papers  (2026-08-31T03:44:54Z)
• SDDMO-Bench: A Benchmark Suite for Streaming Data-Driven Dynamic Multi-Objective Optimization [10.627368862090728]
  This paper proposes SDDMO-Bench, a benchmark suite that transforms dynamic multi-objective test problems into streaming environments.<n>By combining five representative time-dependent base functions with six distributional drift patterns, SDDMO-Bench constructs 30 scenarios.<n> Experiments with representative evolutionary algorithms demonstrate that SDDMO-Bench provides challenging and discriminative test scenarios.
  arXiv  Detail & Related papers  (2026-08-01T06:56:01Z)
• Calibrated Speculative Decoding: Frequency-Guided Candidate Selection for Efficient Inference [27.59556627479635]
  Calibrated Speculative Decoding recovers valid tokens discarded by standard verification.<n>Online Correction Memory aggregates historical rejections to propose recurring divergence patterns as rescue candidates.<n>Semantic Consistency Gating verifies candidate admissibility using probability ratios instead of exact token matching.
  arXiv  Detail & Related papers  (2026-04-15T09:01:54Z)
• SENSE: Semantic Embedding Navigation with Soft-gated Evaluation for Retrieval-based Speculative Decoding [6.448291801598507]
  Speculative Decoding (SD) accelerates Large Language Model (LLM) inference.<n>RSD is favored for its plug-and-play versatility, but its potential is impeded by rigid lexical dependencies.<n>SENSE establishes robust semantic alignment, which empowers the Soft-gated Evaluation module to validate semantic equivalence.
  arXiv  Detail & Related papers  (2026-04-14T04:17:03Z)
• UniDial-EvalKit: A Unified Toolkit for Evaluating Multi-Faceted Conversational Abilities [70.79422099851506]
  We present UniDial-EvalKit (UDE), a unified evaluation toolkit for assessing interactive AI systems.<n>UDE standardizes heterogeneous data formats into a universal schema, streamlines complex evaluation pipelines through a modular architecture, and aligns metric calculations under a consistent scoring interface.
  arXiv  Detail & Related papers  (2026-03-24T13:01:31Z)
• Explainable Semantic Textual Similarity via Dissimilar Span Detection [53.32175252285023]
  We introduce the task of Dissimilar Span Detection (DSD), which aims to identify semantically differing spans between pairs of texts.<n>This can help users understand which particular words or tokens negatively affect the similarity score, or be used to improve performance in STS-dependent downstream tasks.
  arXiv  Detail & Related papers  (2026-03-22T11:32:31Z)
• Utility-Diversity Aware Online Batch Selection for LLM Supervised Fine-tuning [49.04912820721943]
  Supervised fine-tuning (SFT) is computationally expensive and sometimes suffers from overfitting or bias amplification.<n>This work studies the online batch selection family that dynamically scores and filters samples during the training process.<n>We develop textbfUDS (Utility-Diversity Sampling), a framework for efficient online batch selection in SFT.
  arXiv  Detail & Related papers  (2025-10-19T15:32:01Z)
• In-context Ranking Preference Optimization [65.5489745857577]
  We propose an In-context Ranking Preference Optimization (IRPO) framework to optimize large language models (LLMs) based on ranking lists constructed during inference.<n>We show IRPO outperforms standard DPO approaches in ranking performance, highlighting its effectiveness in aligning LLMs with direct in-context ranking preferences.
  arXiv  Detail & Related papers  (2025-04-21T23:06:12Z)
• On the Role of Feedback in Test-Time Scaling of Agentic AI Workflows [71.92083784393418]
  Agentic AI (systems that autonomously plan and act) are becoming widespread, yet their task success rate on complex tasks remains low.<n>Inference-time alignment relies on three components: sampling, evaluation, and feedback.<n>We introduce Iterative Agent Decoding (IAD), a procedure that repeatedly inserts feedback extracted from different forms of critiques.
  arXiv  Detail & Related papers  (2025-04-02T17:40:47Z)
• Beyond the Singular: The Essential Role of Multiple Generations in Effective Benchmark Evaluation and Analysis [10.133537818749291]
  Large language models (LLMs) have demonstrated significant utilities in real-world applications.<n> Benchmark evaluations are crucial for assessing the capabilities of LLMs.
  arXiv  Detail & Related papers  (2025-02-13T03:43:33Z)
• Building Trust in Black-box Optimization: A Comprehensive Framework for Explainability [1.3812010983144802]
  Surrogate Optimization (SO) is a common resolution, yet its proprietary nature leads to a lack of explainability and transparency. We propose emphInclusive Explainability Metrics for Surrogate Optimization (IEMSO) These metrics enhance the transparency, trustworthiness, and explainability of the SO approaches.
  arXiv  Detail & Related papers  (2024-10-18T16:20:17Z)
• A Thorough Performance Benchmarking on Lightweight Embedding-based Recommender Systems [67.52782366565658]
  State-of-the-art recommender systems (RSs) depend on categorical features, which ecoded by embedding vectors, resulting in excessively large embedding tables.<n>Despite the prosperity of lightweight embedding-based RSs, a wide diversity is seen in evaluation protocols.<n>This study investigates various LERS' performance, efficiency, and cross-task transferability via a thorough benchmarking process.
  arXiv  Detail & Related papers  (2024-06-25T07:45:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.