Is Your NPU Ready for LLMs? Dissecting the Hidden Efficiency Bottlenecks in Mobile LLM Inference

• URL: http://arxiv.org/abs/2607.05475v1
• Date: Mon, 06 Jul 2026 09:42:34 GMT
• Title: Is Your NPU Ready for LLMs? Dissecting the Hidden Efficiency Bottlenecks in Mobile LLM Inference
• Abstract summary: Large Language Models (LLMs) on mobile devices enhance privacy and reduce latency, but are bottlenecked by hardware inefficiency.<n>We present the first comprehensive, cross-layer measurement study of mobile LLM inference.<n>We develop PowerBench, a fine-grained profiling tool that provides the first backend-specific energy attribution.
• Score: 8.952888082316456
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deploying Large Language Models (LLMs) on mobile devices enhances privacy and reduces latency, but is severely bottlenecked by hardware inefficiency. We present the first comprehensive, cross-layer measurement study of mobile LLM inference, uniquely spanning five mainstream frameworks (e.g., llama.cpp, GENIE) and three hardware backends (CPU, GPU, NPU). To enable this analysis, we develop PowerBench, a fine-grained profiling tool that provides the first backend-specific energy attribution, moving beyond traditional device-level measurements. Our study yields three critical insights: (1) Framework-induced performance gaps are substantially amplified on NPUs, reaching up to 10x using custom operators due to divergent offloading and quantization strategies. (2) We identify a distinct phase split where NPUs excel at compute-bound prefilling, while CPUs outperform all other backends in memory-bound decoding. This is driven by the NPU's preference for large, fixed-shape workloads, which conflicts with the small-kernel, dynamic nature of decoding. (3) Backend-specific profiling uncovers substantial scheduling headroom missed by prior work. Suboptimal thread configurations, uncoordinated NPU sleep latencies, and CPU polling intervals result in up to 40% energy waste. Leveraging these findings, we present an energy-oriented best-practice configuration for mobile LLM inference. We estimate that this configuration could reduce energy consumption by up to 54.8% on the NPU backend across three datasets.

Related papers

• Efficient On-Device Diffusion LLM Inference with Mobile NPU [8.218014049394391]
  llada is the first NPU-aware inference framework for accelerating dLLMs on smartphones.<n>We implement llada as an end-to-end framework and evaluate it across diverse hardware platforms and dLLM workloads.
  arXiv  Detail & Related papers  (2026-06-11T12:44:57Z)
• When NPUs Are Not Always Faster: A Stage-Level Analysis of Mobile LLM Inference [8.014238682359938]
  Large language models (LLMs) on mobile devices increasingly relies on heterogeneous execution.<n>No prior study has systematically characterized NPU effectiveness at the operator and pipeline level.<n>We present the first stage-aware, multi-level benchmarking study of mobile LLM inference on a CPU-NPU heterogeneous system.
  arXiv  Detail & Related papers  (2026-05-22T10:39:35Z)
• Efficient Mixture-of-Experts LLM Inference with Apple Silicon NPUs [5.431496585727341]
  This paper presents NPUMoE, a runtime inference engine that accelerates MoE execution on Apple Silicon.<n>NPUMoE consistently outperforms baselines, reducing latency by 1.32x-5.55x, improving energy efficiency by 1.81x-7.37x, and reducing CPU-cycle usage by 1.78x-5.54x through effective NPU offloading.
  arXiv  Detail & Related papers  (2026-04-20T19:52:56Z)
• HALO: Semantic-Aware Distributed LLM Inference in Lossy Edge Network [50.33808558714122]
  Large language models' (LLMs) inference at the edge can facilitate prompt service responsiveness while protecting user privacy.<n>We propose HALO, a novel framework that can boost the distributed LLM inference in lossy edge network.<n> Experimental results from a Raspberry Pi cluster demonstrate that HALO achieves a 3.41x end-to-end speedup for LLaMA-series LLMs under unreliable network conditions.
  arXiv  Detail & Related papers  (2026-01-16T07:37:23Z)
• Scaling LLM Test-Time Compute with Mobile NPU on Smartphones [18.50846535848905]
  This paper highlights that mobile Neural Processing Units (NPUs) have underutilized computational resources.<n>We propose applying parallel test-time scaling techniques on mobile NPUs to enhance the performance of smaller LLMs.<n>We show that our approach brings significant speedups: up to 19.0 for mixed-precision GEMM and 2.2 for Softmax.
  arXiv  Detail & Related papers  (2025-09-27T14:17:46Z)
• Fast Matrix Multiplications for Lookup Table-Quantized LLMs [58.11584672945781]
  FLUTE is a flexible lookup table engine for LUT-quantized LLMs.<n>At batch sizes 32 and quantization group size of 128, the FLUTE kernel can be 2-4x faster than existing GEMM kernels.
  arXiv  Detail & Related papers  (2024-07-15T17:55:42Z)
• Fast On-device LLM Inference with NPUs [10.80559106452755]
  We present llm.npu, the first LLM inference system utilizing on-device Neural Processing Unit (NPU) offloading to reduce prefill latency.<n>llm.npu enhances NPU offloading efficiency by re-constructing the prompt and model in three levels.<n>For the first time, llm.npu achieves more than 1,000 tokens/sec prefilling for a billion-sized model.
  arXiv  Detail & Related papers  (2024-07-08T12:20:45Z)
• Enabling High-Sparsity Foundational Llama Models with Efficient Pretraining and Deployment [56.44025052765861]
  Large language models (LLMs) have revolutionized Natural Language Processing (NLP), but their size creates computational bottlenecks. We introduce a novel approach to create accurate, sparse foundational versions of performant LLMs. We show a total speedup on CPUs for sparse-quantized LLaMA models of up to 8.6x.
  arXiv  Detail & Related papers  (2024-05-06T16:03:32Z)
• SqueezeLLM: Dense-and-Sparse Quantization [80.32162537942138]
  Main bottleneck for generative inference with LLMs is memory bandwidth, rather than compute, for single batch inference. We introduce SqueezeLLM, a post-training quantization framework that enables lossless compression to ultra-low precisions of up to 3-bit. Our framework incorporates two novel ideas: (i) sensitivity-based non-uniform quantization, which searches for the optimal bit precision assignment based on second-order information; and (ii) the Dense-and-Sparse decomposition that stores outliers and sensitive weight values in an efficient sparse format.
  arXiv  Detail & Related papers  (2023-06-13T08:57:54Z)
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration [54.692405042065815]
  We propose Activation-aware Weight Quantization (AWQ), a hardware-friendly approach for LLM low-bit weight-only quantization. AWQ protects only 1% salient weights and achieves excellent quantization performance for instruction-tuned LMs and, for the first time, multi-modal LMs. We also implement TinyChat, an efficient and flexible inference framework tailored for 4-bit on-device LLM/VLMs.
  arXiv  Detail & Related papers  (2023-06-01T17:59:10Z)
• Fluid Batching: Exit-Aware Preemptive Serving of Early-Exit Neural Networks on Edge NPUs [74.83613252825754]
  "smart ecosystems" are being formed where sensing happens concurrently rather than standalone. This is shifting the on-device inference paradigm towards deploying neural processing units (NPUs) at the edge. We propose a novel early-exit scheduling that allows preemption at run time to account for the dynamicity introduced by the arrival and exiting processes.
  arXiv  Detail & Related papers  (2022-09-27T15:04:01Z)
• Multi-Exit Semantic Segmentation Networks [78.44441236864057]
  We propose a framework for converting state-of-the-art segmentation models to MESS networks. specially trained CNNs that employ parametrised early exits along their depth to save during inference on easier samples. We co-optimise the number, placement and architecture of the attached segmentation heads, along with the exit policy, to adapt to the device capabilities and application-specific requirements.
  arXiv  Detail & Related papers  (2021-06-07T11:37:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.