MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning

• URL: http://arxiv.org/abs/2110.15352v2
• Date: Wed, 3 Apr 2024 03:12:53 GMT
• Title: MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning
• Authors: Ji Lin, Wei-Ming Chen, Han Cai, Chuang Gan, Song Han,
• Abstract summary: We find that the memory bottleneck is due to the imbalanced memory distribution in convolutional neural network (CNN) designs. We propose a generic patch-by-patch inference scheduling, which significantly cuts down the peak memory. We automate the process with neural architecture search to jointly optimize the neural architecture and inference scheduling, leading to MCUNetV2.
• Score: 72.80896338009579
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Tiny deep learning on microcontroller units (MCUs) is challenging due to the limited memory size. We find that the memory bottleneck is due to the imbalanced memory distribution in convolutional neural network (CNN) designs: the first several blocks have an order of magnitude larger memory usage than the rest of the network. To alleviate this issue, we propose a generic patch-by-patch inference scheduling, which operates only on a small spatial region of the feature map and significantly cuts down the peak memory. However, naive implementation brings overlapping patches and computation overhead. We further propose network redistribution to shift the receptive field and FLOPs to the later stage and reduce the computation overhead. Manually redistributing the receptive field is difficult. We automate the process with neural architecture search to jointly optimize the neural architecture and inference scheduling, leading to MCUNetV2. Patch-based inference effectively reduces the peak memory usage of existing networks by 4-8x. Co-designed with neural networks, MCUNetV2 sets a record ImageNet accuracy on MCU (71.8%), and achieves >90% accuracy on the visual wake words dataset under only 32kB SRAM. MCUNetV2 also unblocks object detection on tiny devices, achieving 16.9% higher mAP on Pascal VOC compared to the state-of-the-art result. Our study largely addressed the memory bottleneck in tinyML and paved the way for various vision applications beyond image classification.

Related papers

• Designing Extremely Memory-Efficient CNNs for On-device Vision Tasks [2.9835839258066015]
  We introduce a memory-efficient CNN (convolutional neural network) for on-device vision tasks. The proposed network classifies ImageNet with extremely low memory (i.e., 63 KB) while achieving competitive top-1 accuracy (i.e., 61.58%) To the best of our knowledge, the memory usage of the proposed network is far smaller than state-of-the-art memory-efficient networks.
  arXiv  Detail & Related papers  (2024-08-07T10:04:04Z)
• Topology-aware Embedding Memory for Continual Learning on Expanding Networks [63.35819388164267]
  We present a framework to tackle the memory explosion problem using memory replay techniques. PDGNNs with Topology-aware Embedding Memory (TEM) significantly outperform state-of-the-art techniques.
  arXiv  Detail & Related papers  (2024-01-24T03:03:17Z)
• MAFAT: Memory-Aware Fusing and Tiling of Neural Networks for Accelerated Edge Inference [1.7894377200944507]
  Machine learning networks can easily exceed available memory, increasing latency due to excessive OS swapping. We propose a memory usage predictor coupled with a search algorithm to provide optimized fusing and tiling configurations. Results show that our approach can run in less than half the memory, and with a speedup of up to 2.78 under severe memory constraints.
  arXiv  Detail & Related papers  (2021-07-14T19:45:49Z)
• Condensation-Net: Memory-Efficient Network Architecture with Cross-Channel Pooling Layers and Virtual Feature Maps [28.992851280809205]
  We propose an algorithm to process a specific network architecture (Condensation-Net) without increasing the maximum memory storage for feature maps. Cross-channel pooling can improve the accuracy of object detection tasks, such as face detection. The overhead to support the cross-channel pooling with the proposed hardware architecture is negligible small.
  arXiv  Detail & Related papers  (2021-04-29T05:44:02Z)
• Robust High-dimensional Memory-augmented Neural Networks [13.82206983716435]
  Memory-augmented neural networks enhance neural networks with an explicit memory to overcome these issues. Access to this explicit memory occurs via soft read and write operations involving every individual memory entry. We propose a robust architecture that employs a computational memory unit as the explicit memory performing analog in-memory computation on high-dimensional (HD) vectors.
  arXiv  Detail & Related papers  (2020-10-05T12:01:56Z)
• Leveraging Automated Mixed-Low-Precision Quantization for tiny edge microcontrollers [76.30674794049293]
  This paper presents an automated mixed-precision quantization flow based on the HAQ framework but tailored for the memory and computational characteristics of MCU devices. Specifically, a Reinforcement Learning agent searches for the best uniform quantization levels, among 2, 4, 8 bits, of individual weight and activation tensors. Given an MCU-class memory bound to 2MB for weight-only quantization, the compressed models produced by the mixed-precision engine result as accurate as the state-of-the-art solutions.
  arXiv  Detail & Related papers  (2020-08-12T06:09:58Z)
• MCUNet: Tiny Deep Learning on IoT Devices [62.752899523628066]
  We propose a framework that jointly designs the efficient neural architecture (TinyNAS) and the lightweight inference engine (TinyEngine) TinyNAS adopts a two-stage neural architecture search approach that first optimize the search space to fit the resource constraints, then specializes the network architecture in the optimized search space. TinyEngine adapts the memory scheduling according to the overall network topology rather than layer-wise optimization, reducing the memory usage by 4.8x.
  arXiv  Detail & Related papers  (2020-07-20T17:59:01Z)
• Efficient Neural Network Deployment for Microcontroller [0.0]
  This paper is going to explore and generalize convolution neural network deployment for microcontrollers. The memory savings and performance will be compared with CMSIS-NN framework developed for ARM Cortex-M CPUs. The final purpose is to develop a tool consuming PyTorch model with trained network weights, and it turns into an optimized inference engine in C/C++ for low memory(kilobyte level) and limited computing capable microcontrollers.
  arXiv  Detail & Related papers  (2020-07-02T19:21:05Z)
• Efficient Integer-Arithmetic-Only Convolutional Neural Networks [87.01739569518513]
  We replace conventional ReLU with Bounded ReLU and find that the decline is due to activation quantization. Our integer networks achieve equivalent performance as the corresponding FPN networks, but have only 1/4 memory cost and run 2x faster on modern GPU.
  arXiv  Detail & Related papers  (2020-06-21T08:23:03Z)
• FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions [70.59851564292828]
  Differentiable Neural Architecture Search (DNAS) has demonstrated great success in designing state-of-the-art, efficient neural networks. We propose a memory and computationally efficient DNAS variant: DMaskingNAS. This algorithm expands the search space by up to $1014times$ over conventional DNAS.
  arXiv  Detail & Related papers  (2020-04-12T08:52:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.