ECHO: Elastic Speculative Decoding with Sparse Gating for High-Concurrency Scenarios

• URL: http://arxiv.org/abs/2604.09603v1
• Date: Tue, 10 Mar 2026 03:51:24 GMT
• Title: ECHO: Elastic Speculative Decoding with Sparse Gating for High-Concurrency Scenarios
• Abstract summary: Speculative Decoding promises to accelerate the inference of Large Language Models, yet its efficacy often degrades in production-grade serving.<n>We introduce ECHO, a high-oriented framework integrated into SGLang that reformulates speculative execution as a budgeted scheduling problem.<n>ECHO consistently outperforms SOTA methods in both low-load and high-load scenarios, achieving up to 5.35x walltime speedup and delivering over 20% relative speedup gain.
• Score: 12.103618967955846
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Speculative Decoding promises to accelerate the inference of Large Language Models, yet its efficacy often degrades in production-grade serving. Existing evaluations typically overlook the compute-bound nature of high-concurrency regimes, where verification compute becomes the dominant bottleneck. Consequently, prior methods face a dilemma: static trees incur massive verification waste, while dynamic trees suffer from cumulative misjudgments and kernel incompatibility. To bridge this gap, we introduce ECHO, a high concurrency-oriented framework integrated into SGLang that reformulates speculative execution as a budgeted scheduling problem. Crucially, ECHO employs sparse confidence gating to manage the batch as a unified super-tree, elastically pivoting budget between depth and width to co-optimize the trade-off between reducing global verification steps and maximizing per-step efficiency. Extensive evaluations across diverse model scales-particularly the industrial-grade Qwen3-235B-demonstrate that ECHO consistently outperforms SOTA methods in both low-load and high-load scenarios, achieving up to 5.35x walltime speedup and delivering over 20% relative speedup gain.

Related papers

• ECHO: Early-layer Collaborative Hierarchical Orchestration with Bonus Logits in Speculative Decoding [60.34475274870224]
  ECHO is a hierarchical dual-loop framework that exploits the functional asymmetry between LLM layers.<n>Within the inner loop, early-layer bonus logits drive rapid, multi-step draft-tree exploration at a minimal cost.<n>ECHO significantly boosts mean accepted tokens and achieves a 2.4$times$ to 2.9$times$ speedup.
  arXiv  Detail & Related papers  (2026-09-15T14:21:33Z)
• Trees from Marginals: Autoregressive drafting with factorized priors [0.0]
  Speculative decoding greatly increases the interactivity of autoregressive language models.<n>We introduce Weaver, a lightweight autoregressive adapter that constructs proposal trees from the top-K marginals of a factorized drafter.
  arXiv  Detail & Related papers  (2026-07-07T19:48:36Z)
• Dual-Dimensional Consistency: Balancing Budget and Quality in Adaptive Inference-Time Scaling [12.49034574686497]
  Large Language Models (LLMs) have demonstrated remarkable abilities in reasoning.<n> maximizing their potential through inference-time scaling faces challenges in trade-off between sampling budget and reasoning quality.<n>We propose Dual-Dimensional Consistency (DDC), a unified framework that bridges path quality with adaptive termination.
  arXiv  Detail & Related papers  (2026-05-14T17:19:37Z)
• Universally Empowering Zeroth-Order Optimization via Adaptive Layer-wise Sampling [43.822941944402544]
  Zeroth-Order optimization presents a promising memory-efficient paradigm for fine-tuning Large Language Models.<n>However, its practical adoption is severely constrained by slow wall-clock convergence and high estimation variance.<n>We propose AdaLeZO, an Adaptive Layer-wise ZO optimization framework.
  arXiv  Detail & Related papers  (2026-04-20T13:37:31Z)
• Tackling Length Inflation Without Trade-offs: Group Relative Reward Rescaling for Reinforcement Learning [74.5532558466687]
  Group Relative Reward Rescaling (GR$3$) is a general, continuous, and reward-dependent gating mechanism.<n> GR$3$maintains training dynamics and downstream performance comparable to standard GRPO.<n>It significantly mitigates length inflation, outperforming state-of-the-art length-regularized baselines.
  arXiv  Detail & Related papers  (2026-03-11T08:41:34Z)
• Adaptive-Horizon Conflict-Based Search for Closed-Loop Multi-Agent Path Finding [10.628717510399046]
  MAPF is a core coordination problem for large robot fleets in automated warehouses and logistics.<n>Existing approaches are either open-loop planners, which generate fixed trajectories, or closed-loops without reliable performance guarantees.<n>This paper presents ACCBS, a closed-loop built on a finite-horizon variant of CBS with a horizon-changing mechanism.
  arXiv  Detail & Related papers  (2026-02-12T14:55:16Z)
• HQP: Sensitivity-Aware Hybrid Quantization and Pruning for Ultra-Low-Latency Edge AI Inference [0.0]
  Hybrid Quantization and Pruning (HQP) framework designed to achieve synergistic model acceleration.<n>HQP framework achieves a peak performance gain of 3.12 times inference speedup and a 55 percent model size reduction.
  arXiv  Detail & Related papers  (2026-02-02T18:17:45Z)
• Towards Efficient Agents: A Co-Design of Inference Architecture and System [66.59916327634639]
  This paper presents AgentInfer, a unified framework for end-to-end agent acceleration.<n>We decompose the problem into four synergistic components: AgentCollab, AgentSched, AgentSAM, and AgentCompress.<n>Experiments on the BrowseComp-zh and DeepDiver benchmarks demonstrate that through the synergistic collaboration of these methods, AgentInfer reduces ineffective token consumption by over 50%.
  arXiv  Detail & Related papers  (2025-12-20T12:06:13Z)
• READER: Retrieval-Assisted Drafter for Efficient LLM Inference [0.0386965802948046]
  Autoregressive Language Models instantiate a factorized likelihood over token sequences, yet their strictly sequential decoding process imposes an intrinsic lower bound on latency inference.<n>This bottleneck has emerged as a central obstacle to the scalable deployment of large-scale generative models.<n>We present READER, a speculative decoding framework that bypasses the training of the auxiliary draft model.
  arXiv  Detail & Related papers  (2025-08-12T16:47:48Z)
• Beyond Fixed: Training-Free Variable-Length Denoising for Diffusion Large Language Models [74.15250326312179]
  Diffusion Large Language Models offer efficient parallel generation and capable global modeling.<n>The dominant application ofDLLMs is hindered by the need for a statically predefined generation length.<n>We introduce DAEDAL, a novel training-free denoising strategy that enables Dynamic Adaptive Length Expansion.
  arXiv  Detail & Related papers  (2025-08-01T17:56:07Z)
• Generalized Linear Bandits: Almost Optimal Regret with One-Pass Update [70.38810219913593]
  We study the generalized linear bandit (GLB) problem, a contextual multi-armed bandit framework that extends the classical linear model by incorporating a non-linear link function.<n>GLBs are widely applicable to real-world scenarios, but their non-linear nature introduces significant challenges in achieving both computational and statistical efficiency.<n>We propose a jointly efficient algorithm that attains a nearly optimal regret bound with $mathcalO(1)$ time and space complexities per round.
  arXiv  Detail & Related papers  (2025-07-16T02:24:21Z)
• MPQ-DMv2: Flexible Residual Mixed Precision Quantization for Low-Bit Diffusion Models with Temporal Distillation [74.34220141721231]
  We present MPQ-DMv2, an improved textbfMixed textbfPrecision textbfQuantization framework for extremely low-bit textbfDiffusion textbfModels.
  arXiv  Detail & Related papers  (2025-07-06T08:16:50Z)
• Accelerating Chain-of-Thought Reasoning: When Goal-Gradient Importance Meets Dynamic Skipping [3.521097198612099]
  Adaptive GoGI-Skip is a novel framework learning dynamic CoT compression via supervised fine-tuning.<n>It achieves substantial efficiency gains - reducing CoT token counts by over 45% on average and delivering 1.6-2.0 times inference speedups.<n> Notably, it significantly outperforms existing baselines by preserving accuracy even at high effective compression rates.
  arXiv  Detail & Related papers  (2025-05-13T09:39:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.