Efficient Mixture-of-Experts LLM Inference with Apple Silicon NPUs

• URL: http://arxiv.org/abs/2604.18788v1
• Date: Mon, 20 Apr 2026 19:52:56 GMT
• Title: Efficient Mixture-of-Experts LLM Inference with Apple Silicon NPUs
• Abstract summary: This paper presents NPUMoE, a runtime inference engine that accelerates MoE execution on Apple Silicon.<n>NPUMoE consistently outperforms baselines, reducing latency by 1.32x-5.55x, improving energy efficiency by 1.81x-7.37x, and reducing CPU-cycle usage by 1.78x-5.54x through effective NPU offloading.
• Score: 5.431496585727341
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Apple Neural Engine (ANE) is a dedicated neural processing unit (NPU) present in every Apple Silicon chip. Mixture-of-Experts (MoE) LLMs improve inference efficiency via sparse activation but are challenging for NPUs in three ways: expert routing is unpredictable and introduces dynamic tensor shapes that conflict with the shape-specific constraints of NPUs; several irregular operators, e.g., top-k, scatter/gather, etc., are not NPU-friendly; and launching many small expert kernels incurs substantial dispatch and synchronization overhead. NPUs are designed to offload AI compute from CPU and GPU; our goal is to enable such offloading for MoE inference, particularly during prefill, where long-context workloads consume substantial system resources. This paper presents NPUMoE, a runtime inference engine that accelerates MoE execution on Apple Silicon by offloading dense, static computation to NPU, while preserving a CPU/GPU fallback path for dynamic operations. NPUMoE uses offline calibration to estimate expert capacity and popularity that drives three key techniques: (1) Static tiers for expert capacity to address dynamic expert routing (2) Grouped expert execution to mitigate NPU concurrency limits (3) Load-aware expert compute graph residency to reduce CPU-NPU synchronization overhead. Experiments on Apple M-series devices using three representative MoE LLMs and four long-context workloads show that NPUMoE consistently outperforms baselines, reducing latency by 1.32x-5.55x, improving energy efficiency by 1.81x-7.37x, and reducing CPU-cycle usage by 1.78x-5.54x through effective NPU offloading.

Related papers

• Is Your NPU Ready for LLMs? Dissecting the Hidden Efficiency Bottlenecks in Mobile LLM Inference [8.952888082316456]
  Large Language Models (LLMs) on mobile devices enhance privacy and reduce latency, but are bottlenecked by hardware inefficiency.<n>We present the first comprehensive, cross-layer measurement study of mobile LLM inference.<n>We develop PowerBench, a fine-grained profiling tool that provides the first backend-specific energy attribution.
  arXiv  Detail & Related papers  (2026-07-06T09:42:34Z)
• Efficient On-Device Diffusion LLM Inference with Mobile NPU [8.218014049394391]
  llada is the first NPU-aware inference framework for accelerating dLLMs on smartphones.<n>We implement llada as an end-to-end framework and evaluate it across diverse hardware platforms and dLLM workloads.
  arXiv  Detail & Related papers  (2026-06-11T12:44:57Z)
• When NPUs Are Not Always Faster: A Stage-Level Analysis of Mobile LLM Inference [8.014238682359938]
  Large language models (LLMs) on mobile devices increasingly relies on heterogeneous execution.<n>No prior study has systematically characterized NPU effectiveness at the operator and pipeline level.<n>We present the first stage-aware, multi-level benchmarking study of mobile LLM inference on a CPU-NPU heterogeneous system.
  arXiv  Detail & Related papers  (2026-05-22T10:39:35Z)
• ENEC: A Lossless AI Model Compression Method Enabling Fast Inference on Ascend NPUs [13.980477697764014]
  ENEC is a novel compression method specifically customized for AI model weights and optimized for Ascend Neural Processing Units.<n>Compared to leading GPU solutions, ENEC achieves a 3.43X higher throughput than DietGPU and a 1.12X better compression ratio than nvCOMP.<n>ENEC significantly improves end-to-end inference performance, achieving up to a 6.3X speedup.
  arXiv  Detail & Related papers  (2026-03-28T16:11:56Z)
• Evaluating the Energy Efficiency of NPU-Accelerated Machine Learning Inference on Embedded Microcontrollers [0.0]
  This paper evaluates the impact of Neural Processing Units (NPUs) on machine learning (ML) execution on microcontrollers (MCUs)<n>It shows substantial efficiency gains when inference is offloaded to the NPU.<n>For moderate to large networks, latency improvements ranged from 7x to over 125x, with per-inference net energy reductions up to 143x.
  arXiv  Detail & Related papers  (2025-09-22T08:52:54Z)
• eIQ Neutron: Redefining Edge-AI Inference with Integrated NPU and Compiler Innovations [4.776283807742058]
  eIQ Neutron efficient-NPU is integrated into a commercial flagship MPU.<n>Our solution achieves an average speedup of 1.8x (4x peak) at equal TOPS and memory resources across standard AI-benchmarks.
  arXiv  Detail & Related papers  (2025-09-17T19:45:51Z)
• Intra-DP: A High Performance Collaborative Inference System for Mobile Edge Computing [67.98609858326951]
  Intra-DP is a high-performance collaborative inference system optimized for deep neural networks (DNNs) on mobile devices.<n>It reduces per-inference latency by up to 50% and energy consumption by up to 75% compared to state-of-the-art baselines.<n>The evaluation demonstrates that Intra-DP reduces per-inference latency by up to 50% and energy consumption by up to 75% compared to state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-07-08T09:50:57Z)
• MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models [58.3342517278868]
  This paper describes the design of Mixed-precision AutoRegressive LINear kernels. It shows that batchsizes up to 16-32 can be supported with close to maximum ($4times$) quantization speedup. MarLIN accomplishes this via a combination of techniques, such as asynchronous memory access, complex task scheduling and pipelining.
  arXiv  Detail & Related papers  (2024-08-21T16:10:41Z)
• Fast On-device LLM Inference with NPUs [10.80559106452755]
  We present llm.npu, the first LLM inference system utilizing on-device Neural Processing Unit (NPU) offloading to reduce prefill latency.<n>llm.npu enhances NPU offloading efficiency by re-constructing the prompt and model in three levels.<n>For the first time, llm.npu achieves more than 1,000 tokens/sec prefilling for a billion-sized model.
  arXiv  Detail & Related papers  (2024-07-08T12:20:45Z)
• Latency-aware Unified Dynamic Networks for Efficient Image Recognition [72.8951331472913]
  LAUDNet is a framework to bridge the theoretical and practical efficiency gap in dynamic networks. It integrates three primary dynamic paradigms-spatially adaptive computation, dynamic layer skipping, and dynamic channel skipping. It can notably reduce the latency of models like ResNet by over 50% on platforms such as V100,3090, and TX2 GPUs.
  arXiv  Detail & Related papers  (2023-08-30T10:57:41Z)
• Intelligence Processing Units Accelerate Neuromorphic Learning [52.952192990802345]
  Spiking neural networks (SNNs) have achieved orders of magnitude improvement in terms of energy consumption and latency. We present an IPU-optimized release of our custom SNN Python package, snnTorch.
  arXiv  Detail & Related papers  (2022-11-19T15:44:08Z)
• Fluid Batching: Exit-Aware Preemptive Serving of Early-Exit Neural Networks on Edge NPUs [74.83613252825754]
  "smart ecosystems" are being formed where sensing happens concurrently rather than standalone. This is shifting the on-device inference paradigm towards deploying neural processing units (NPUs) at the edge. We propose a novel early-exit scheduling that allows preemption at run time to account for the dynamicity introduced by the arrival and exiting processes.
  arXiv  Detail & Related papers  (2022-09-27T15:04:01Z)
• An Adaptive Device-Edge Co-Inference Framework Based on Soft Actor-Critic [72.35307086274912]
  High-dimension parameter model and large-scale mathematical calculation restrict execution efficiency, especially for Internet of Things (IoT) devices. We propose a new Deep Reinforcement Learning (DRL)-Soft Actor Critic for discrete (SAC-d), which generates the emphexit point, emphexit point, and emphcompressing bits by soft policy iterations. Based on the latency and accuracy aware reward design, such an computation can well adapt to the complex environment like dynamic wireless channel and arbitrary processing, and is capable of supporting the 5G URL
  arXiv  Detail & Related papers  (2022-01-09T09:31:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.