APEX: Adaptive Expert Prefetching for Memory-Efficient Edge MoE Inference

• URL: http://arxiv.org/abs/2608.11688v1
• Date: Wed, 12 Aug 2026 05:57:09 GMT
• Title: APEX: Adaptive Expert Prefetching for Memory-Efficient Edge MoE Inference
• Abstract summary: Mixture-of-Experts (MoE) models are attractive for edge deployment because they provide high model capacity while activating only a small subset of parameters per token.<n>We present APEX: Adaptive Expert Prefetching, a predictive resource management framework that overlaps expert loading with useful computation.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Mixture-of-Experts (MoE) models are attractive for edge deployment because they provide high model capacity while activating only a small subset of parameters per token, improving compute efficiency. However, MoE inference at the edge is fundamentally limited by memory. Expert parameters are large and often reside in off-chip memory due to capacity, cost, and power constraints, putting expert loading to the critical path. We present APEX: Adaptive Expert Prefetching, a predictive resource management framework that overlaps expert loading with useful computation. APEX introduces a lightweight prefetch router that predicts candidate experts before the attention block to dynamically fetch additional experts using a learned confidence model. This adaptive strategy achieves over 99% overlap accuracy, significantly outperforming fixed top-k prefetching techniques. APEX supports two execution modes: a correctness-preserving mode that guarantees exact routing semantics, and a stall-free mode that eliminates residual stalls by operating on available experts with negligible impact on application accuracy. Across multiple MoE models, the correctness-preserving mode reduces per-token latency by up to 26% and improves energy-delay product (EDP) by up to 41% over state-of-the-art baselines, while the stall-free mode provides additional efficiency gains with negligible impact on application accuracy. These results establish adaptive, confidence-driven expert prefetching as an effective approach for efficient MoE inference on edge systems.

Related papers

• Attribution-Guided and Coverage-Maximized Pruning for Structural MoE Compression [74.00650541246374]
  We propose a structural pruning framework tailored for MoE models.<n>Our approach reduces memory footprint by 5.27$times and consistently outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-06-16T06:53:27Z)
• BEAM: Binary Expert Activation Masking for Dynamic Routing in MoE [18.898045833395095]
  Mixture-of-Experts (MoE) architectures activate only a subset of experts per token.<n>Existing acceleration methods require costly retraining with architectural changes or suffer from severe performance drop at high sparsity.<n>We propose BEAM, a novel method that learns token-adaptive expert selection via trainable binary masks.
  arXiv  Detail & Related papers  (2026-05-14T06:33:41Z)
• DyMoE: Dynamic Expert Orchestration with Mixed-Precision Quantization for Efficient MoE Inference on Edge [49.3949520091098]
  We propose DyMoE, a dynamic mixed-precision quantization framework for high-performance edge inference.<n>We show that DyMoE reduces Time-to-First-Token by 3.44x-22.7x and up to a 14.58x speedup in Time-Per-Output-Token (TPOT) compared to state-of-the-art offloading baselines.
  arXiv  Detail & Related papers  (2026-03-19T17:30:01Z)
• FineRMoE: Dimension Expansion for Finer-Grained Expert with Its Upcycling Approach [58.345210583013454]
  FineRMoE is an architecture that extends fine-grained expert design to both intermediate and output dimensions.<n>It achieves 6 times higher parameter efficiency, 281 times lower prefill latency, and 136 timese higher decoding throughput during inference.
  arXiv  Detail & Related papers  (2026-03-09T12:00:36Z)
• MC#: Mixture Compressor for Mixture-of-Experts Large Models [86.64315380917827]
  Mixture-of-Experts (MoE) effectively scales large language models (LLMs) and vision-language models (VLMs) by increasing capacity through sparse activation.<n>We propose MC# (Mixture-Compressor-sharp), a framework that combines static quantization and dynamic expert pruning.
  arXiv  Detail & Related papers  (2025-10-13T03:12:46Z)
• LExI: Layer-Adaptive Active Experts for Efficient MoE Model Inference [2.8653469160349077]
  We introduce LExI, a data-free optimization technique that determines the optimal number of active experts per layer in a pretrained MoE model.<n>Experiments on state-of-the-art language and vision MoE benchmarks demonstrate that LExI significantly outperforms traditional MoE pruning approaches in terms of inference efficiency.
  arXiv  Detail & Related papers  (2025-09-02T19:01:46Z)
• HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference [54.40808356999408]
  We present HOBBIT, a mixed precision expert offloading system to enable flexible and efficient MoE inference. Our key insight is that dynamically replacing less critical cache-miss experts with low precision versions can substantially reduce expert-loading latency. HOBBIT achieves up to a 9.93x speedup in decoding compared to state-of-the-art MoE offloading systems.
  arXiv  Detail & Related papers  (2024-11-03T04:25:46Z)
• ExpertFlow: Optimized Expert Activation and Token Allocation for Efficient Mixture-of-Experts Inference [41.41316718220569]
  ExpertFlow is designed to enhance inference efficiency by accommodating flexible routing and enabling efficient expert scheduling between CPU and GPU. Our experiments demonstrate that ExpertFlow achieves up to 93.72% GPU memory savings and enhances inference speed by 2 to 10 times compared to baseline methods.
  arXiv  Detail & Related papers  (2024-10-23T15:24:54Z)
• AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inference [13.263938935671646]
  AdapMoE is an algorithm-system co-design framework for efficient MoE inference. AdapMoE features adaptive expert gating and management to reduce the on-demand loading overheads. We show AdapMoE consistently outperforms existing techniques, reducing the average number of activated experts by 25% and achieving a 1.35x speedup without degradation accuracy.
  arXiv  Detail & Related papers  (2024-08-19T03:27:15Z)
• SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Experts [49.01990048827639]
  We introduce SEER-MoE, a framework for reducing both the memory footprint and compute requirements of pre-trained MoE models. The first stage involves pruning the total number of experts using a heavy-hitters counting guidance, while the second stage employs a regularization-based fine-tuning strategy to recover accuracy loss. Our empirical studies demonstrate the effectiveness of our method, resulting in a sparse MoEs model optimized for inference efficiency with minimal accuracy trade-offs.
  arXiv  Detail & Related papers  (2024-04-07T22:13:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.