Experience-Driven Dynamic Exits for LLMs with Reinforcement Learning

• URL: http://arxiv.org/abs/2606.03113v1
• Date: Tue, 02 Jun 2026 03:59:14 GMT
• Title: Experience-Driven Dynamic Exits for LLMs with Reinforcement Learning
• Abstract summary: LEDE learns a policy to dynamically select the optimal exit layer and speculation length based on the local context.<n> Comprehensive evaluations on Llama-2 and Llama-3 models show LEDE achieves up to a $2.0times$$sim$$2.7times$ speedup over autoregressive decoding.
• Score: 16.19290569626178
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models suffer from slow autoregressive inference. While self-speculative decoding accelerates this process, its efficiency is hampered by static configurations like fixed exit layers and speculation lengths. We reframe this optimization as a \textbf{Markov Decision Process} and propose \textbf{LEDE}, a framework that uses offline reinforcement learning. LEDE learns a policy to dynamically select the optimal exit layer and speculation length based on the local context of the generated sequence at each step, balancing computational cost and draft quality. Comprehensive evaluations on Llama-2 and Llama-3 models show LEDE achieves up to a $2.0\times$$\sim$$2.7\times$ speedup over autoregressive decoding and and provides an additional 17\% speedup over the static speculative baselines.

Related papers

• FlexEE: Self-Speculative and KV-Compatible Early Exiting for Offloading-Aware LLM Inference [6.9575993729793595]
  Large language model (LLM) inference is often constrained by both computation and memory.<n>We present FlexEE, an early exiting framework for resource-constrained and offloading-based LLM inference.
  arXiv  Detail & Related papers  (2026-09-15T11:20:23Z)
• WIDE: Boosting Adaptive LLM Inference via Token-level Dynamic Width Pruning [12.767703956755751]
  We present the first end-to-end differentiable token-level dynamic width pruning framework for both prefill and decode scenarios.<n>We propose a pruning- kernel co-design framework that decomposes dynamic sparsity acceleration into mask reordering, hardware-agnostic block-level skipping, and hardware-dependent intra-block skipping.
  arXiv  Detail & Related papers  (2026-07-30T16:01:03Z)
• Learning-Augmented Scalable Linear Assignment Problem Optimization via Neural Dual Warm-Starts [19.540758462427878]
  We propose a learning-augmented framework that accelerates exact assignment solvers while maintaining optimality and worst-case guarantees.<n>We introduce RowDualNet, a lightweight row-independent architecture that avoids the $mathcalO(N2)$ memory bottleneck of graph-based models.
  arXiv  Detail & Related papers  (2026-05-10T07:15:49Z)
• AutoSP: Unlocking Long-Context LLM Training Via Compiler-Based Sequence Parallelism [14.600392553338681]
  Large-language-models (LLMs) demonstrate enormous utility in long-context tasks.<n>Existing LLM training libraries do not provide easy to use abstractions to optimize for long-context training.<n>We introduce AutoSP: the first automated solution to automatically optimize LLM training for longer-contexts.
  arXiv  Detail & Related papers  (2026-04-29T18:31:29Z)
• Flux Attention: Context-Aware Hybrid Attention for Efficient LLMs Inference [36.81958522560282]
  We introduce Flux Attention, a context-aware framework that dynamically optimize attention at the layer level.<n>As a parameter-efficient approach, our framework requires only 12 hours of training on 8$times$A800 GPUs.<n>Our framework achieves a superior trade-off between performance and inference speed compared with baseline models, with speed improvements of up to $2.8times$ and $2.0times$ in the prefill and decode stages.
  arXiv  Detail & Related papers  (2026-04-08T07:36:17Z)
• $\nabla$-Reasoner: LLM Reasoning via Test-Time Gradient Descent in Latent Space [71.23672814629448]
  $nabla$-Reasoner is an iterative generation framework that integrates differentiable optimization over token logits into the decoding loop.<n>$nabla$-Reasoner achieves over 20% accuracy improvement on a challenging mathematical reasoning benchmark.
  arXiv  Detail & Related papers  (2026-03-05T08:42:54Z)
• KnapSpec: Self-Speculative Decoding via Adaptive Layer Selection as a Knapsack Problem [12.668341559890605]
  We propose KnapSpec, a training-free framework that reformulates draft model selection as a knapsack problem to maximize tokens-per-time throughput.<n>We provide the first rigorous theoretical analysis establishing cosine similarity between hidden states as a mathematically sound proxy for the token acceptance rate.<n>Our experiments on Qwen3 and Llama3 demonstrate that KnapSpec consistently outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-02-23T08:13:03Z)
• CAS-Spec: Cascade Adaptive Self-Speculative Decoding for On-the-Fly Lossless Inference Acceleration of LLMs [48.8252978488871]
  Speculative decoding offers seamless integration and broad utility when deploying large language models.<n>Cascading a hierarchy of draft models promises further acceleration and flexibility, but the high cost of training multiple models has limited its practical application.<n>We propose a novel Cascade Adaptive Self-Speculative Decoding (CAS-Spec) method which constructs speculative draft models.
  arXiv  Detail & Related papers  (2025-10-30T08:51:29Z)
• Beyond Fixed: Training-Free Variable-Length Denoising for Diffusion Large Language Models [74.15250326312179]
  Diffusion Large Language Models offer efficient parallel generation and capable global modeling.<n>The dominant application ofDLLMs is hindered by the need for a statically predefined generation length.<n>We introduce DAEDAL, a novel training-free denoising strategy that enables Dynamic Adaptive Length Expansion.
  arXiv  Detail & Related papers  (2025-08-01T17:56:07Z)
• Pangu Embedded: An Efficient Dual-system LLM Reasoner with Metacognition [95.54406667705999]
  Pangu Embedded is an efficient Large Language Model (LLM) reasoner developed on Ascend Neural Processing Units (NPUs)<n>It addresses the significant computational costs and inference latency challenges prevalent in existing reasoning-optimized LLMs.<n>It delivers rapid responses and state-of-the-art reasoning quality within a single, unified model architecture.
  arXiv  Detail & Related papers  (2025-05-28T14:03:02Z)
• Towards Efficient Automatic Self-Pruning of Large Language Models [55.90119819642064]
  Post-training structured pruning is a promising solution that prunes Large Language Models without the need for retraining.<n>We argue that the key to mitigating this issue lies in accurately determining the pruning rate for each layer.<n>We introduce $textbfSelf-Pruner$ an end-to-end automatic self-pruning framework for LLMs, which efficiently search layer-wise pruning rates.
  arXiv  Detail & Related papers  (2025-02-20T09:59:50Z)
• COrAL: Order-Agnostic Language Modeling for Efficient Iterative Refinement [80.18490952057125]
  Iterative refinement has emerged as an effective paradigm for enhancing the capabilities of large language models (LLMs) on complex tasks. We propose Context-Wise Order-Agnostic Language Modeling (COrAL) to overcome these challenges. Our approach models multiple token dependencies within manageable context windows, enabling the model to perform iterative refinement internally.
  arXiv  Detail & Related papers  (2024-10-12T23:56:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.