Benchmarking Ultra-Low-Power $μ$NPUs

• URL: http://arxiv.org/abs/2503.22567v3
• Date: Fri, 31 Oct 2025 02:19:39 GMT
• Title: Benchmarking Ultra-Low-Power $μ$NPUs
• Abstract summary: Efficient on-device neural network (NN) inference offers predictable latency, improved privacy and reliability, and lower operating costs for vendors.<n>Recently, microcontroller-scale NN accelerators, also known as neural processing units ($mu$NPUs), have been developed for ultra-low-power applications.<n>We present the first comparative evaluation of a number of commercially-available $mu$NPUs, including the first independent benchmarks for multiple platforms.
• Score: 5.1032153291137465
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Efficient on-device neural network (NN) inference offers predictable latency, improved privacy and reliability, and lower operating costs for vendors than cloud-based inference. This has sparked recent development of microcontroller-scale NN accelerators, also known as neural processing units ($\mu$NPUs), designed specifically for ultra-low-power applications. We present the first comparative evaluation of a number of commercially-available $\mu$NPUs, including the first independent benchmarks for multiple platforms. To ensure fairness, we develop and open-source a model compilation pipeline supporting consistent benchmarking of quantized models across diverse microcontroller hardware. Our resulting analysis uncovers both expected performance trends as well as surprising disparities between hardware specifications and actual performance, including certain $\mu$NPUs exhibiting unexpected scaling behaviors with model complexity. This work provides a foundation for ongoing evaluation of $\mu$NPU platforms, alongside offering practical insights for both hardware and software developers in this rapidly evolving space.

Related papers

• Cloud to Edge: Benchmarking LLM Inference On Hardware-Accelerated Single-Board Computers [0.5249805590164902]
  Large language models (LLMs) are becoming increasingly capable at small parameter scales.<n>Existing LLM-specific edge benchmarking efforts rely on CPU-only inference.<n>This paper proposes a multi-dimensional benchmarking methodology that jointly evaluates inference performance and hardware efficiency.
  arXiv  Detail & Related papers  (2026-04-24T14:57:57Z)
• Option Pricing on Noisy Intermediate-Scale Quantum Computers: A Quantum Neural Network Approach [0.0]
  We propose a fully quantum approach to option pricing based on Quantum Neural Networks (QNNs)<n>We present one of the first implementations of such a methodology on currently available quantum hardware.<n>Our implementation utilizes a compact 2-qubit QNN architecture evaluated across multiple state-of-the-art quantum processors.
  arXiv  Detail & Related papers  (2026-04-20T23:03:57Z)
• Scaling LLM Test-Time Compute with Mobile NPU on Smartphones [18.50846535848905]
  This paper highlights that mobile Neural Processing Units (NPUs) have underutilized computational resources.<n>We propose applying parallel test-time scaling techniques on mobile NPUs to enhance the performance of smaller LLMs.<n>We show that our approach brings significant speedups: up to 19.0 for mixed-precision GEMM and 2.2 for Softmax.
  arXiv  Detail & Related papers  (2025-09-27T14:17:46Z)
• Task-Oriented Real-time Visual Inference for IoVT Systems: A Co-design Framework of Neural Networks and Edge Deployment [61.20689382879937]
  Task-oriented edge computing addresses this by shifting data analysis to the edge. Existing methods struggle to balance high model performance with low resource consumption. We propose a novel co-design framework to optimize neural network architecture.
  arXiv  Detail & Related papers  (2024-10-29T19:02:54Z)
• Temporal Feature Matters: A Framework for Diffusion Model Quantization [105.3033493564844]
  Diffusion models rely on the time-step for the multi-round denoising.<n>We introduce a novel quantization framework that includes three strategies.<n>This framework preserves most of the temporal information and ensures high-quality end-to-end generation.
  arXiv  Detail & Related papers  (2024-07-28T17:46:15Z)
• Enhancing Dropout-based Bayesian Neural Networks with Multi-Exit on FPGA [20.629635991749808]
  This paper proposes an algorithm and hardware co-design framework that can generate field-programmable gate array (FPGA)-based accelerators for efficient BayesNNs. At the algorithm level, we propose novel multi-exit dropout-based BayesNNs with reduced computational and memory overheads. At the hardware level, this paper introduces a transformation framework that can generate FPGA-based accelerators for the proposed efficient BayesNNs.
  arXiv  Detail & Related papers  (2024-06-20T17:08:42Z)
• Few-Shot Testing: Estimating Uncertainty of Memristive Deep Neural Networks Using One Bayesian Test Vector [0.0]
  We propose a test vector generation framework that can estimate the model uncertainty of NNs implemented on memristor-based CIM hardware. Our method is evaluated on different model dimensions, tasks, fault rates, and variation noise to show that it can consistently achieve $100%$ coverage with only $0.024$ MB of memory overhead.
  arXiv  Detail & Related papers  (2024-05-29T08:53:16Z)
• Cheaply Evaluating Inference Efficiency Metrics for Autoregressive Transformer APIs [66.30706841821123]
  Large language models (LLMs) power many state-of-the-art systems in natural language processing. LLMs are extremely computationally expensive, even at inference time. We propose a new metric for comparing inference efficiency across models.
  arXiv  Detail & Related papers  (2023-05-03T21:51:42Z)
• Fluid Batching: Exit-Aware Preemptive Serving of Early-Exit Neural Networks on Edge NPUs [74.83613252825754]
  "smart ecosystems" are being formed where sensing happens concurrently rather than standalone. This is shifting the on-device inference paradigm towards deploying neural processing units (NPUs) at the edge. We propose a novel early-exit scheduling that allows preemption at run time to account for the dynamicity introduced by the arrival and exiting processes.
  arXiv  Detail & Related papers  (2022-09-27T15:04:01Z)
• MAPLE-X: Latency Prediction with Explicit Microprocessor Prior Knowledge [87.41163540910854]
  Deep neural network (DNN) latency characterization is a time-consuming process. We propose MAPLE-X which extends MAPLE by incorporating explicit prior knowledge of hardware devices and DNN architecture latency.
  arXiv  Detail & Related papers  (2022-05-25T11:08:20Z)
• Towards Optimal VPU Compiler Cost Modeling by using Neural Networks to Infer Hardware Performances [58.720142291102135]
  'VPUNN' is a neural network-based cost model trained on low-level task profiling. It consistently outperforms the state-of-the-art cost modeling in Intel's line of VPU processors.
  arXiv  Detail & Related papers  (2022-05-09T22:48:39Z)
• MAPLE: Microprocessor A Priori for Latency Estimation [81.91509153539566]
  Modern deep neural networks must demonstrate state-of-the-art accuracy while exhibiting low latency and energy consumption. Measuring the latency of every evaluated architecture adds a significant amount of time to the NAS process. We propose Microprocessor A Priori for Estimation Estimation MAPLE that does not rely on transfer learning or domain adaptation.
  arXiv  Detail & Related papers  (2021-11-30T03:52:15Z)
• FLASH: Fast Neural Architecture Search with Hardware Optimization [7.263481020106725]
  Neural architecture search (NAS) is a promising technique to design efficient and high-performance deep neural networks (DNNs) This paper proposes FLASH, a very fast NAS methodology that co-optimizes the DNN accuracy and performance on a real hardware platform.
  arXiv  Detail & Related papers  (2021-08-01T23:46:48Z)
• ANNETTE: Accurate Neural Network Execution Time Estimation with Stacked Models [56.21470608621633]
  We propose a time estimation framework to decouple the architectural search from the target hardware. The proposed methodology extracts a set of models from micro- kernel and multi-layer benchmarks and generates a stacked model for mapping and network execution time estimation. We compare estimation accuracy and fidelity of the generated mixed models, statistical models with the roofline model, and a refined roofline model for evaluation.
  arXiv  Detail & Related papers  (2021-05-07T11:39:05Z)
• Generalized Latency Performance Estimation for Once-For-All Neural Architecture Search [0.0]
  We introduce two generalizability strategies which include fine-tuning using a base model trained on a specific hardware and NAS search space. We provide a family of latency prediction models that achieve over 50% lower RMSE loss as compared to ProxylessNAS.
  arXiv  Detail & Related papers  (2021-01-04T00:48:09Z)
• HAPI: Hardware-Aware Progressive Inference [18.214367595727037]
  Convolutional neural networks (CNNs) have recently become the state-of-the-art in a diversity of AI tasks. Despite their popularity, CNN inference still comes at a high computational cost. This work presents HAPI, a novel methodology for generating high-performance early-exit networks.
  arXiv  Detail & Related papers  (2020-08-10T09:55:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.