FlexEE: Self-Speculative and KV-Compatible Early Exiting for Offloading-Aware LLM Inference

• URL: http://arxiv.org/abs/2609.17008v1
• Date: Tue, 15 Sep 2026 11:20:23 GMT
• Title: FlexEE: Self-Speculative and KV-Compatible Early Exiting for Offloading-Aware LLM Inference
• Abstract summary: Large language model (LLM) inference is often constrained by both computation and memory.<n>We present FlexEE, an early exiting framework for resource-constrained and offloading-based LLM inference.
• Score: 6.9575993729793595
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language model (LLM) inference is often constrained by both computation and memory, especially in offloading-based deployments where model weights are transferred across memory hierarchies during autoregressive decoding. In this setting, reducing the number of executed layers can lower per-token latency while also avoiding costly weight movement. Motivated by this observation, we present FlexEE, an early exiting framework for resource-constrained and offloading-based LLM inference. FlexEE makes early exiting practical for LLM decoding through layer-wise exit supervision for reliable intermediate-layer prediction, self-speculative decoding over a Top-K local vocabulary for low-cost exit decisions, and dynamic hidden state management for KV-cache-correct and memory-aware execution. Across generative and downstream tasks, FlexEE enables efficient early exit with minimal accuracy degradation, delivering up to 1.27$\times$/3.16$\times$ and 1.25$\times$/2.83$\times$ end-to-end speedups on Llama2-7B and Llama3-8B under 0\%/50\% weight offloading, respectively.

Related papers

• LLM Inference in a Flash! [66.43896490006965]
  Large Language Models (LLMs) have shown impressive capabilities across a range of natural language processing tasks.<n>We present an end-to-end integer-only quantization approach to eliminate expensive floating-point computations.
  arXiv  Detail & Related papers  (2026-09-14T18:04:30Z)
• Experience-Driven Dynamic Exits for LLMs with Reinforcement Learning [16.19290569626178]
  LEDE learns a policy to dynamically select the optimal exit layer and speculation length based on the local context.<n> Comprehensive evaluations on Llama-2 and Llama-3 models show LEDE achieves up to a $2.0times$$sim$$2.7times$ speedup over autoregressive decoding.
  arXiv  Detail & Related papers  (2026-06-02T03:59:14Z)
• MemDLM: Memory-Enhanced DLM Training [56.40248490616793]
  Diffusion Language Models (DLMs) offer attractive advantages over Auto-Regressive (AR) models.<n>They suffer from a notable train-inference mismatch.<n>We propose MemDLM (Memory-Enhanced DLM), which embeds a simulated denoising process into training.
  arXiv  Detail & Related papers  (2026-03-23T17:39:56Z)
• Fast Forward: Accelerating LLM Prefill with Predictive FFN Sparsity [5.223181756178433]
  We introduce FastForward, a predictive sparsity framework that accelerates large language model (LLM) prefill through block-wise, context-aware FFN sparsity.<n>FastForward delivers up to 1.45$times$ compute-bound speedup at 50% FFN sparsity with $$ 6% accuracy loss compared to the dense baseline on LongBench.
  arXiv  Detail & Related papers  (2026-01-30T23:23:40Z)
• SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruning [3.502168555273189]
  SlimInfer aims to accelerate inference by directly pruning less critical prompt tokens during the forward pass.<n>We show that SlimInfer can achieve up to $mathbf2.53times$ time-to-first-token (TTFT) speedup and $mathbf1.88times$ end-to-end latency reduction for LLaMA3.1-8B-Instruct.
  arXiv  Detail & Related papers  (2025-08-08T16:42:38Z)
• Sparse-dLLM: Accelerating Diffusion LLMs with Dynamic Cache Eviction [72.27673320976933]
  Diffusion Large Language Models (dLLMs) enable breakthroughs in reasoning and parallel decoding.<n>Current caching techniques accelerate decoding by storing full-layer states, yet impose substantial memory usage.<n>We propose Sparse-dLLM, the first training-free framework integrating dynamic cache eviction with sparse attention.
  arXiv  Detail & Related papers  (2025-08-04T16:14:03Z)
• BucketServe: Bucket-Based Dynamic Batching for Smart and Efficient LLM Inference Serving [3.620158146761518]
  BucketServe is a bucket-based dynamic framework designed to optimize inference performance.<n>It can handle 1.93x more request load load attainment of 80% compared with UELLM and demonstrates 1.975x higher system load capacity compared to the UELLM.
  arXiv  Detail & Related papers  (2025-07-23T01:51:48Z)
• Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding [51.711605076319216]
  Diffusion-based large language models (Diffusion LLMs) have shown promise for non-autoregressive text generation with parallel decoding capabilities.<n>We introduce a novel block-wise approximate KV Cache mechanism tailored for bidirectional diffusion models, enabling cache reuse with negligible performance drop.<n>We propose a confidence-aware parallel decoding strategy that selectively decodes tokens exceeding a confidence threshold, mitigating dependency violations and maintaining generation quality.
  arXiv  Detail & Related papers  (2025-05-28T17:39:15Z)
• TrimLLM: Progressive Layer Dropping for Domain-Specific LLMs [11.615399679746675]
  Specializing large language models (LLMs) for local deployment in domain-specific use cases is necessary for strong performance.<n>We develop TrimLLM based on the layer-wise specialization phenomenon we empirically observed and verified on contemporary LLMs.<n>We show it retains LLMs' capacity in specific domains and inference speedup achieves irrespective of hardware and deep learning frameworks.
  arXiv  Detail & Related papers  (2024-12-15T16:47:16Z)
• EoRA: Fine-tuning-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximation [84.70637613266835]
  EoRA is a fine-tuning-free method that augments compressed Large Language Models with low-rank matrices.<n>EoRA consistently outperforms prior training-free low rank methods in recovering the accuracy of compressed LLMs.
  arXiv  Detail & Related papers  (2024-10-28T17:59:03Z)
• FFN-SkipLLM: A Hidden Gem for Autoregressive Decoding with Adaptive Feed Forward Skipping [49.66872823080736]
  Autoregressive Large Language Models (e.g., LLaMa, GPTs) are omnipresent achieving remarkable success in language understanding and generation. To mitigate overload incurred during generation, several early-exit and layer-dropping strategies have been proposed. We propose FFN-SkipLLM, which is an input-adaptive feed-forward skipping strategy.
  arXiv  Detail & Related papers  (2024-04-05T02:35:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.