EAGLE-Pangu: Accelerator-Safe Tree Speculative Decoding on Ascend NPUs

• URL: http://arxiv.org/abs/2603.08088v1
• Date: Mon, 09 Mar 2026 08:30:04 GMT
• Title: EAGLE-Pangu: Accelerator-Safe Tree Speculative Decoding on Ascend NPUs
• Abstract summary: We present a system that ports tree speculative decoding to a Pangu teacher backend on Ascend NPUs.<n>Eagles-Pangu improves end-to-end decoding throughput by 1.27x on average, up to 2.46x at p99.
• Score: 5.19049674298842
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Autoregressive decoding remains a primary bottleneck in large language model (LLM) serving, motivating speculative decoding methods that reduce expensive teacher-model invocations by verifying multiple candidate tokens per step. Tree-structured speculation further increases parallelism, but is often brittle when ported across heterogeneous backends and accelerator stacks, where attention masking, KV-cache layouts, and indexing semantics are not interchangeable. We present EAGLE-Pangu, a reproducible system that ports EAGLE-3-style tree speculative decoding to a Pangu teacher backend on Ascend NPUs. EAGLE-Pangu contributes (i) an explicit branch/commit cache manager built on the Cache API, (ii) accelerator-safe tree tensorization that removes undefined negative indices by construction and validates structural invariants, and (iii) a fused-kernel-compatible teacher verification path with a debuggable eager fallback. On 240 turns from MT-Bench and HumanEval-style prompts, EAGLE-Pangu improves end-to-end decoding throughput by 1.27x on average, up to 2.46x at p99, over teacher-only greedy decoding in the fused-kernel performance path. We also provide a fused-kernel-free reference path with structured traces and invariant checks to support reproducible debugging and ablation across execution modes and tree budgets.

Related papers

• Accelerating Unified Multimodal Models with Core-Expansion Routing and Unified Computation Scheduling [61.28599393707943]
  Unified multimodal models jointly support understanding and generation, but incur substantial redundant computation across tokens, layers, and generation timesteps.<n>We propose a task-shared core scorer and progress-conditioned generation expansions, optimized through generation decomposition and cross-task core alignment.<n> Experiments on two representative UMM architectures demonstrate consistent quality--efficiency improvements across both tasks, retaining 98.03% of dense understanding performance with a 1.93times$ end-to-end inference speedup.
  arXiv  Detail & Related papers  (2026-08-29T14:27:55Z)
• DEPT: Document Embedding Preservation Tuning for Unified Query Expansion and Retrieval [45.09142622162023]
  Large language models (LLMs) can both expand underspecified queries and encode text as dense representations.<n>We train a single decoder-only LLM end to end, where the same model generates the expansion and encodes both the expanded query and candidate documents.<n>We introduce Document Embedding Preservation Tuning (DEPT), which keeps tuned document embeddings close to cached initial embeddings.
  arXiv  Detail & Related papers  (2026-08-18T10:52:08Z)
• Bridging the Structural Gap: Adapting Autoregressive Generation for Recommendation [10.616703360070359]
  Generative Recommendation (GR) has emerged as a new paradigm for sequential recommendation.<n>We propose BARGE, which employs Item Context-Aware Attention (ICA) to restore item-level structure during encoding.<n>Experiments and analytical studies on public benchmarks and a large-scale offline test demonstrate that BARGE achieves superior recommendation performance.
  arXiv  Detail & Related papers  (2026-07-23T08:11:57Z)
• DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation [69.8840101036735]
  Speculative decoding accelerates Large Language Model (LLM) inference.<n>Recent parallel drafters efficiently propose long token sequences in a single forward pass, but suffer from rapid acceptance decay due to a lack of inter-token dependencies.<n>We introduce DSpark, a speculative decoding framework that unifies high- throughput parallel generation with adaptive, load-aware verification.
  arXiv  Detail & Related papers  (2026-07-06T14:28:06Z)
• Prefix-Adaptive Block Diffusion for Efficient Document Recognition [52.15352911463151]
  Block Diffusion Models (BDMs) support parallel generation, flexible-length output, and KV caching, making them promising for efficient document parsing.<n>We propose the Prefix- Block Diffusion Model (PA-BDM) which replaces intra-block bidirectional denoising with causal denoising from prefix to suffix.<n> Experiments show that the 3B PA-BDM achieves higher recognition scores on several benchmarks and improves inference throughput by 71.6% over the 2.5B MinerU-Diffusion.
  arXiv  Detail & Related papers  (2026-05-16T07:50:13Z)
• APCD: Adaptive Path-Contrastive Decoding for Reliable Large Language Model Generation [9.34156158035883]
  Large language models (LLMs) often suffer from hallucinations due to error accumulation in autoregressive decoding.<n>We propose Adaptive Path-Contrastive Decoding (APCD), a multi-path decoding framework that improves output reliability through adaptive exploration and controlled path interaction.
  arXiv  Detail & Related papers  (2026-05-10T11:57:39Z)
• Beyond Scattered Acceptance: Fast and Coherent Inference for DLMs via Longest Stable Prefixes [10.877713536966601]
  Longestahead Prefix (LSP) scheduler is a training-free and model-agnostic inference paradigm based on monolithic prefix absorption.<n>LSP evaluates token stability via a single forward pass, dynamically identifies a contiguous left-aligned block of stable predictions.<n>It snaps its boundary to natural linguistic or structural acceptances before an atomic commitment.
  arXiv  Detail & Related papers  (2026-03-05T18:25:26Z)
• WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference [44.87788417755154]
  We propose WeDLM, a diffusion decoding framework built entirely on standard causal attention.<n>We show that WeDLM preserves the quality of strong AR backbones while delivering substantial speedups.
  arXiv  Detail & Related papers  (2025-12-28T01:25:48Z)
• FastEagle: Cascaded Drafting for Accelerating Speculative Decoding [6.482154864678126]
  We present FastEagle, a non-autoregressive cascaded drafter that emits an entire draft in a single forward pass.<n>FastEagle delivers substantial wall-clock speedups over strong autoregressive drafters while maintaining competitive acceptance behavior.
  arXiv  Detail & Related papers  (2025-09-24T09:38:32Z)
• cAST: Enhancing Code Retrieval-Augmented Generation with Structural Chunking via Abstract Syntax Tree [39.50252992647112]
  Retrieval-Augmented Generation (RAG) has become essential for large-scale code generation.<n>Our work highlights the importance of structure-aware chunking for scaling retrieval-enhanced code intelligence.
  arXiv  Detail & Related papers  (2025-06-18T17:31:51Z)
• ProtInvTree: Deliberate Protein Inverse Folding with Reward-guided Tree Search [77.55575655986252]
  ProtInvTree is a reward-guided tree-search framework for protein inverse folding.<n>It reformulates sequence generation as a deliberate, step-wise decision-making process.<n>It supports flexible test-time scaling by expanding the search depth and breadth without retraining.
  arXiv  Detail & Related papers  (2025-06-01T09:34:20Z)
• Pangu Embedded: An Efficient Dual-system LLM Reasoner with Metacognition [95.54406667705999]
  Pangu Embedded is an efficient Large Language Model (LLM) reasoner developed on Ascend Neural Processing Units (NPUs)<n>It addresses the significant computational costs and inference latency challenges prevalent in existing reasoning-optimized LLMs.<n>It delivers rapid responses and state-of-the-art reasoning quality within a single, unified model architecture.
  arXiv  Detail & Related papers  (2025-05-28T14:03:02Z)
• ASAG: Building Strong One-Decoder-Layer Sparse Detectors via Adaptive Sparse Anchor Generation [50.01244854344167]
  We bridge the performance gap between sparse and dense detectors by proposing Adaptive Sparse Anchor Generator (ASAG) ASAG predicts dynamic anchors on patches rather than grids in a sparse way so that it alleviates the feature conflict problem. Our method outperforms dense-d ones and achieves a better speed-accuracy trade-off.
  arXiv  Detail & Related papers  (2023-08-18T02:06:49Z)
• Quick Dense Retrievers Consume KALE: Post Training Kullback Leibler Alignment of Embeddings for Asymmetrical dual encoders [89.29256833403169]
  We introduce Kullback Leibler Alignment of Embeddings (KALE), an efficient and accurate method for increasing the inference efficiency of dense retrieval methods. KALE extends traditional Knowledge Distillation after bi-encoder training, allowing for effective query encoder compression without full retraining or index generation. Using KALE and asymmetric training, we can generate models which exceed the performance of DistilBERT despite having 3x faster inference.
  arXiv  Detail & Related papers  (2023-03-31T15:44:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.