MNN: A Universal and Efficient Inference Engine

• URL: http://arxiv.org/abs/2002.12418v1
• Date: Thu, 27 Feb 2020 20:03:16 GMT
• Title: MNN: A Universal and Efficient Inference Engine
• Authors: Xiaotang Jiang, Huan Wang, Yiliu Chen, Ziqi Wu, Lichuan Wang, Bin Zou, Yafeng Yang, Zongyang Cui, Yu Cai, Tianhang Yu, Chengfei Lv, Zhihua Wu
• Abstract summary: Mobile Neural Network (MNN) is a universal and efficient inference engine tailored to mobile applications. The contributions of MNN include: (1) presenting a mechanism called pre-inference that manages to conduct runtime optimization; (2)deliveringthorough kernel optimization on operators to achieve optimal performance; (3) introducing backend abstraction module which enables hybrid scheduling and keeps the engine lightweight.
• Score: 6.830174586230231
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deploying deep learning models on mobile devices draws more and more attention recently. However, designing an efficient inference engine on devices is under the great challenges of model compatibility, device diversity, and resource limitation. To deal with these challenges, we propose Mobile Neural Network (MNN), a universal and efficient inference engine tailored to mobile applications. In this paper, the contributions of MNN include: (1) presenting a mechanism called pre-inference that manages to conduct runtime optimization; (2)deliveringthorough kernel optimization on operators to achieve optimal computation performance; (3) introducing backend abstraction module which enables hybrid scheduling and keeps the engine lightweight. Extensive benchmark experiments demonstrate that MNN performs favorably against other popular lightweight deep learning frameworks. MNN is available to public at: https://github.com/alibaba/MNN.

Related papers

• MNN-LLM: A Generic Inference Engine for Fast Large Language Model Deployment on Mobile Devices [4.385815629175844]
  MNN-LLM is a framework specifically designed to accelerate the deployment of large language models on mobile devices.<n>It addresses the runtime characteristics of LLMs through model quantization and DRAM-Flash hybrid storage.<n> Notably, MNN-LLM achieves up to a 8.6x speed increase compared to current mainstream LLM-specific frameworks.
  arXiv  Detail & Related papers  (2025-06-12T07:45:29Z)
• Scalable Mechanistic Neural Networks [52.28945097811129]
  We propose an enhanced neural network framework designed for scientific machine learning applications involving long temporal sequences. By reformulating the original Mechanistic Neural Network (MNN) we reduce the computational time and space complexities from cubic and quadratic with respect to the sequence length, respectively, to linear. Extensive experiments demonstrate that S-MNN matches the original MNN in precision while substantially reducing computational resources.
  arXiv  Detail & Related papers  (2024-10-08T14:27:28Z)
• A Multi-Head Ensemble Multi-Task Learning Approach for Dynamical Computation Offloading [62.34538208323411]
  We propose a multi-head ensemble multi-task learning (MEMTL) approach with a shared backbone and multiple prediction heads (PHs) MEMTL outperforms benchmark methods in both the inference accuracy and mean square error without requiring additional training data.
  arXiv  Detail & Related papers  (2023-09-02T11:01:16Z)
• Resource Constrained Model Compression via Minimax Optimization for Spiking Neural Networks [11.19282454437627]
  Spiking Neural Networks (SNNs) have the characteristics of event-driven and high energy-efficient networks. It is difficult to deploy these networks on resource-limited edge devices directly. We propose an improved end-to-end Minimax optimization method for this sparse learning problem.
  arXiv  Detail & Related papers  (2023-08-09T02:50:15Z)
• Adaptive DNN Surgery for Selfish Inference Acceleration with On-demand Edge Resource [25.274288063300844]
  Deep Neural Networks (DNNs) have significantly improved the accuracy of intelligent applications on mobile devices. DNN surgery can enable real-time inference despite the computational limitations of mobile devices. This paper introduces a novel Decentralized DNN Surgery (DDS) framework.
  arXiv  Detail & Related papers  (2023-06-21T11:32:28Z)
• E3NE: An End-to-End Framework for Accelerating Spiking Neural Networks with Emerging Neural Encoding on FPGAs [6.047137174639418]
  End-to-end framework E3NE automates the generation of efficient SNN inference logic for FPGAs. E3NE uses less than 50% of hardware resources and 20% less power, while reducing the latency by an order of magnitude.
  arXiv  Detail & Related papers  (2021-11-19T04:01:19Z)
• GRIM: A General, Real-Time Deep Learning Inference Framework for Mobile Devices based on Fine-Grained Structured Weight Sparsity [46.75304109970339]
  This paper designs a novel mobile inference acceleration framework GRIM that is General to both convolutional neural networks (CNNs) and recurrent neural networks (RNNs) We propose a new fine-grained structured sparsity scheme through the Block-based Column-Row (BCR) pruning. Based on this new fine-grained structured sparsity, our GRIM framework consists of two parts: (a) the compiler optimization and code generation for real-time mobile inference.
  arXiv  Detail & Related papers  (2021-08-25T03:50:46Z)
• Towards Real-Time DNN Inference on Mobile Platforms with Model Pruning and Compiler Optimization [56.3111706960878]
  High-end mobile platforms serve as primary computing devices for a wide range of Deep Neural Network (DNN) applications. constrained computation and storage resources on these devices pose significant challenges for real-time inference executions. We propose a set of hardware-friendly structured model pruning and compiler optimization techniques to accelerate DNN executions on mobile devices.
  arXiv  Detail & Related papers  (2020-04-22T03:18:23Z)
• The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understanding [97.85957811603251]
  We present MT-DNN, an open-source natural language understanding (NLU) toolkit that makes it easy for researchers and developers to train customized deep learning models. Built upon PyTorch and Transformers, MT-DNN is designed to facilitate rapid customization for a broad spectrum of NLU tasks. A unique feature of MT-DNN is its built-in support for robust and transferable learning using the adversarial multi-task learning paradigm.
  arXiv  Detail & Related papers  (2020-02-19T03:05:28Z)
• PatDNN: Achieving Real-Time DNN Execution on Mobile Devices with Pattern-based Weight Pruning [57.20262984116752]
  We introduce a new dimension, fine-grained pruning patterns inside the coarse-grained structures, revealing a previously unknown point in design space. With the higher accuracy enabled by fine-grained pruning patterns, the unique insight is to use the compiler to re-gain and guarantee high hardware efficiency.
  arXiv  Detail & Related papers  (2020-01-01T04:52:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.