Hardware-Assisted Virtualization of Neural Processing Units for Cloud Platforms

• URL: http://arxiv.org/abs/2408.04104v3
• Date: Fri, 13 Sep 2024 02:48:33 GMT
• Title: Hardware-Assisted Virtualization of Neural Processing Units for Cloud Platforms
• Abstract summary: We present Neu10, a holistic NPU virtualization framework. Neu10 consists of (1) a flexible NPU abstraction called vNPU, which enables fine-grained virtualization of the heterogeneous compute units in a physical NPU (pNPU); (2) a vNPU resource allocator that enables pay-as-you-go computing model and flexible vNPU-to-pNPU mappings for improved resource utilization and cost-effectiveness; and (3) an ISA extension of modern NPU architecture for facilitating fine-grained tensor operator scheduling for multiple vNPUs.
• Score: 3.93151902626205
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Cloud platforms today have been deploying hardware accelerators like neural processing units (NPUs) for powering machine learning (ML) inference services. To maximize the resource utilization while ensuring reasonable quality of service, a natural approach is to virtualize NPUs for efficient resource sharing for multi-tenant ML services. However, virtualizing NPUs for modern cloud platforms is not easy. This is not only due to the lack of system abstraction support for NPU hardware, but also due to the lack of architectural and ISA support for enabling fine-grained dynamic operator scheduling for virtualized NPUs. We present Neu10, a holistic NPU virtualization framework. We investigate virtualization techniques for NPUs across the entire software and hardware stack. Neu10 consists of (1) a flexible NPU abstraction called vNPU, which enables fine-grained virtualization of the heterogeneous compute units in a physical NPU (pNPU); (2) a vNPU resource allocator that enables pay-as-you-go computing model and flexible vNPU-to-pNPU mappings for improved resource utilization and cost-effectiveness; (3) an ISA extension of modern NPU architecture for facilitating fine-grained tensor operator scheduling for multiple vNPUs. We implement Neu10 based on a production-level NPU simulator. Our experiments show that Neu10 improves the throughput of ML inference services by up to 1.4$\times$ and reduces the tail latency by up to 4.6$\times$, while improving the NPU utilization by 1.2$\times$ on average, compared to state-of-the-art NPU sharing approaches.

Related papers

• DynaNDE: Dynamic Near-Data Expert Scheduling for Batched MoE Inference [1.3714927666270709]
  Mixture-of-Experts (MoE) models suffer from substantial data-movement overhead when deployed on neural processing unit (NPU)-based systems.<n>NDP provides a promising way to mitigate this bottleneck via cooperative NPU-NDP execution.<n>This paper presents DynaNDE, a dynamic near-data expert scheduling framework that exploits NPU-NDP collaboration to accelerate batched MoE inference.
  arXiv  Detail & Related papers  (2026-08-31T21:38:59Z)
• When NPUs Are Not Always Faster: A Stage-Level Analysis of Mobile LLM Inference [8.014238682359938]
  Large language models (LLMs) on mobile devices increasingly relies on heterogeneous execution.<n>No prior study has systematically characterized NPU effectiveness at the operator and pipeline level.<n>We present the first stage-aware, multi-level benchmarking study of mobile LLM inference on a CPU-NPU heterogeneous system.
  arXiv  Detail & Related papers  (2026-05-22T10:39:35Z)
• Efficient Mixture-of-Experts LLM Inference with Apple Silicon NPUs [5.431496585727341]
  This paper presents NPUMoE, a runtime inference engine that accelerates MoE execution on Apple Silicon.<n>NPUMoE consistently outperforms baselines, reducing latency by 1.32x-5.55x, improving energy efficiency by 1.81x-7.37x, and reducing CPU-cycle usage by 1.78x-5.54x through effective NPU offloading.
  arXiv  Detail & Related papers  (2026-04-20T19:52:56Z)
• eIQ Neutron: Redefining Edge-AI Inference with Integrated NPU and Compiler Innovations [4.776283807742058]
  eIQ Neutron efficient-NPU is integrated into a commercial flagship MPU.<n>Our solution achieves an average speedup of 1.8x (4x peak) at equal TOPS and memory resources across standard AI-benchmarks.
  arXiv  Detail & Related papers  (2025-09-17T19:45:51Z)
• FeNN: A RISC-V vector processor for Spiking Neural Network acceleration [0.1350479308585481]
  Spiking Neural Networks (SNNs) have the potential to drastically reduce the energy requirements of AI systems.<n>Here, we present a novel RISC-V-based soft vector processor (FeNN) tailored to SNN simulating on FPGAs.
  arXiv  Detail & Related papers  (2025-06-13T13:13:54Z)
• Benchmarking Ultra-Low-Power $μ$NPUs [5.577830976560545]
  This paper presents the first comparative evaluation of a number of commercially-available $mu$NPUs. We develop and open-source a model compilation framework to enable consistent benchmarking of quantized models across diverse $mu$NPU hardware.
  arXiv  Detail & Related papers  (2025-03-28T16:14:06Z)
• Flex-TPU: A Flexible TPU with Runtime Reconfigurable Dataflow Architecture [0.0]
  The work herein consists of developing a reconfigurable dataflow TPU, called the Flex-TPU, which can dynamically change the dataflow per layer during run-time. The results show that our Flex-TPU design achieves a significant performance increase of up to 2.75x compared to conventional TPU, with only minor area and power overheads.
  arXiv  Detail & Related papers  (2024-07-11T17:33:38Z)
• Intelligence Processing Units Accelerate Neuromorphic Learning [52.952192990802345]
  Spiking neural networks (SNNs) have achieved orders of magnitude improvement in terms of energy consumption and latency. We present an IPU-optimized release of our custom SNN Python package, snnTorch.
  arXiv  Detail & Related papers  (2022-11-19T15:44:08Z)
• TPU-MLIR: A Compiler For TPU Using MLIR [2.6519283973116963]
  TPU-MLIR deploys pre-trained neural network (NN) models to a custom ASIC called a Processing Unit (TPU) A NN model is translated to the TOP dialect and then lowered to the TPU dialect for different TPUs according to the chip's configuration. We demonstrate how to use the MLIR pass pipeline to organize and perform optimization on TPU to generate machine code.
  arXiv  Detail & Related papers  (2022-10-23T10:45:54Z)
• Fluid Batching: Exit-Aware Preemptive Serving of Early-Exit Neural Networks on Edge NPUs [74.83613252825754]
  "smart ecosystems" are being formed where sensing happens concurrently rather than standalone. This is shifting the on-device inference paradigm towards deploying neural processing units (NPUs) at the edge. We propose a novel early-exit scheduling that allows preemption at run time to account for the dynamicity introduced by the arrival and exiting processes.
  arXiv  Detail & Related papers  (2022-09-27T15:04:01Z)
• MAPLE-X: Latency Prediction with Explicit Microprocessor Prior Knowledge [87.41163540910854]
  Deep neural network (DNN) latency characterization is a time-consuming process. We propose MAPLE-X which extends MAPLE by incorporating explicit prior knowledge of hardware devices and DNN architecture latency.
  arXiv  Detail & Related papers  (2022-05-25T11:08:20Z)
• U-Boost NAS: Utilization-Boosted Differentiable Neural Architecture Search [50.33956216274694]
  optimizing resource utilization in target platforms is key to achieving high performance during DNN inference. We propose a novel hardware-aware NAS framework that does not only optimize for task accuracy and inference latency, but also for resource utilization. We achieve 2.8 - 4x speedup for DNN inference compared to prior hardware-aware NAS methods.
  arXiv  Detail & Related papers  (2022-03-23T13:44:15Z)
• An Adaptive Device-Edge Co-Inference Framework Based on Soft Actor-Critic [72.35307086274912]
  High-dimension parameter model and large-scale mathematical calculation restrict execution efficiency, especially for Internet of Things (IoT) devices. We propose a new Deep Reinforcement Learning (DRL)-Soft Actor Critic for discrete (SAC-d), which generates the emphexit point, emphexit point, and emphcompressing bits by soft policy iterations. Based on the latency and accuracy aware reward design, such an computation can well adapt to the complex environment like dynamic wireless channel and arbitrary processing, and is capable of supporting the 5G URL
  arXiv  Detail & Related papers  (2022-01-09T09:31:50Z)
• Compiling ONNX Neural Network Models Using MLIR [51.903932262028235]
  We present a preliminary report on our onnx-mlir compiler, which generates code for the inference of deep neural network models. Onnx-mlir relies on the Multi-Level Intermediate Representation (MLIR) infrastructure recently integrated in the LLVM project.
  arXiv  Detail & Related papers  (2020-08-19T05:28:08Z)
• Dopant Network Processing Units: Towards Efficient Neural-network Emulators with High-capacity Nanoelectronic Nodes [0.0]
  "Dopant Network Processing Units" (DNPUs) are highly energy-efficient and have potentially very high throughput. We introduce DNPUs as high-capacity neurons and move from a single to a multi-neuron framework. We show that feed-forward DNPU networks improve the performance of a single DNPU from 77% to 94% test accuracy.
  arXiv  Detail & Related papers  (2020-07-24T06:35:44Z)
• Fully-parallel Convolutional Neural Network Hardware [0.7829352305480285]
  We propose a new power-and-area-efficient architecture for implementing Articial Neural Networks (ANNs) in hardware. For the first time, a fully-parallel CNN as LENET-5 is embedded and tested in a single FPGA.
  arXiv  Detail & Related papers  (2020-06-22T17:19:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.