ECHO: Early-layer Collaborative Hierarchical Orchestration with Bonus Logits in Speculative Decoding

• URL: http://arxiv.org/abs/2609.17241v1
• Date: Tue, 15 Sep 2026 14:21:33 GMT
• Title: ECHO: Early-layer Collaborative Hierarchical Orchestration with Bonus Logits in Speculative Decoding
• Abstract summary: ECHO is a hierarchical dual-loop framework that exploits the functional asymmetry between LLM layers.<n>Within the inner loop, early-layer bonus logits drive rapid, multi-step draft-tree exploration at a minimal cost.<n>ECHO significantly boosts mean accepted tokens and achieves a 2.4$times$ to 2.9$times$ speedup.
• Score: 60.34475274870224
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While draft-model-free speculative decoding offers a promising path to efficient LLM inference, it is frequently constrained by stale draft candidates and the high computational cost of the verification. To address these challenges, we propose ECHO, a hierarchical dual-loop framework that exploits the functional asymmetry between LLM layers. Leveraging the high discriminative efficiency of early layers and the authoritative distribution of final layers, ECHO bifurcates inference into a high-frequency inner loop and a low-frequency outer loop. Within the inner loop, early-layer bonus logits drive rapid, multi-step draft-tree exploration at a minimal cost. Simultaneously, the outer loop performs authoritative full-model verification through a state-reuse mechanism. Crucially, the outer loop also utilizes final-layer bonus logits to correct existing paths and supplement the tree with high-confidence candidates for subsequent cycles. Experimental results across diverse benchmarks demonstrate that ECHO significantly boosts mean accepted tokens and achieves a 2.4$\times$ to 2.9$\times$ speedup, outperforming existing state-of-the-art baselines with negligible engineering overhead and no extra deployment parameters, albeit with a one-shot fine-tuning dependency for optimal acceleration. The code is available at https://github.com/whucs21Mzy/ECHO.

Related papers

• Flash-BoN: Instant Drafts for Inference-Time Scaling in Diffusion Models [91.19049123903137]
  Inference-time scaling for text-to-image generation has progressed from simple Best-of-$N$ sampling to guided search methods.<n>These approaches focus on when and how often to verify during denoising but largely treat the cost of generation itself as fixed.<n>We show that under wall-clock evaluation, simple BoN already matches or outperforms several guided search techniques.
  arXiv  Detail & Related papers  (2026-07-05T19:02:32Z)
• Context Pruning for Coding Agents via Multi-Rubric Latent Reasoning [14.508049757681862]
  LLM-powered coding agents spend the majority of their token budget reading repository files.<n>We show that this formulation creates a modeling bottleneck.<n>We propose LaMR, a structured pruning framework that decomposes code relevance into two interpretable quality dimensions.
  arXiv  Detail & Related papers  (2026-05-14T18:30:10Z)
• ECHO: Elastic Speculative Decoding with Sparse Gating for High-Concurrency Scenarios [12.103618967955846]
  Speculative Decoding promises to accelerate the inference of Large Language Models, yet its efficacy often degrades in production-grade serving.<n>We introduce ECHO, a high-oriented framework integrated into SGLang that reformulates speculative execution as a budgeted scheduling problem.<n>ECHO consistently outperforms SOTA methods in both low-load and high-load scenarios, achieving up to 5.35x walltime speedup and delivering over 20% relative speedup gain.
  arXiv  Detail & Related papers  (2026-03-10T03:51:24Z)
• FineRMoE: Dimension Expansion for Finer-Grained Expert with Its Upcycling Approach [58.345210583013454]
  FineRMoE is an architecture that extends fine-grained expert design to both intermediate and output dimensions.<n>It achieves 6 times higher parameter efficiency, 281 times lower prefill latency, and 136 timese higher decoding throughput during inference.
  arXiv  Detail & Related papers  (2026-03-09T12:00:36Z)
• $\nabla$-Reasoner: LLM Reasoning via Test-Time Gradient Descent in Latent Space [71.23672814629448]
  $nabla$-Reasoner is an iterative generation framework that integrates differentiable optimization over token logits into the decoding loop.<n>$nabla$-Reasoner achieves over 20% accuracy improvement on a challenging mathematical reasoning benchmark.
  arXiv  Detail & Related papers  (2026-03-05T08:42:54Z)
• Pipeline Parallelism is All You Need for Optimized Early-Exit Based Self-Speculative Decoding [73.67253077506672]
  Large language models (LLMs) deliver impressive generation quality, but incur very high inference cost.<n>Early-exit based self-speculative decoding (EESD) has emerged to mitigate this cost.<n>We propose Pipeline-Parallel Self-Speculative Decoding (PPSD) that fully pipelines the draft and verification work.
  arXiv  Detail & Related papers  (2025-09-19T04:51:41Z)
• SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applications [13.948608558319307]
  Speculative decoding is widely adopted to reduce latency in large language model (LLM) inference.<n>Agentic frameworks submit repetitive inference requests, which result in long and highly predictable computation.<n>We introduce emphSuffixDecoding, a novel method that utilizes efficient suffix trees to cache long token sequences.
  arXiv  Detail & Related papers  (2024-11-07T18:49:33Z)
• Non-stationary Delayed Online Convex Optimization: From Full-information to Bandit Setting [71.82716109461967]
  We propose an algorithm called Mild-OGD for the full-information case, where delayed gradients are available.<n>We show that the dynamic regret of Mild-OGD can be automatically bounded by $O(sqrtbardT(P_T+1))$ under the in-order assumption.<n>We also develop a bandit variant of Mild-OGD for a more challenging case with only delayed loss values.
  arXiv  Detail & Related papers  (2023-05-20T07:54:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.