Split CNN Inference on Networked Microcontrollers

• URL: http://arxiv.org/abs/2605.09357v1
• Date: Sun, 10 May 2026 06:16:11 GMT
• Title: Split CNN Inference on Networked Microcontrollers
• Abstract summary: Deep neural networks on microcontrollers (MCUs) are severely constrained by limited memory resources.<n>TinyML techniques fail in practice due to excessive peak Random Access Memory (RAM) usage during inference.<n>We present a fine-grained split inference system for networked MCUs that enables collaborative inference of Convolutional Neural Networks (CNN) models across multiple devices.
• Score: 13.922830929023783
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Running deep neural networks on microcontroller units (MCUs) is severely constrained by limited memory resources. While TinyML techniques reduce model size and computation, they often fail in practice due to excessive peak Random Access Memory (RAM) usage during inference, dominated by intermediate activations. As a result, many models remain infeasible on standalone MCUs. In this work, we present a fine-grained split inference system for networked MCUs that enables collaborative inference of Convolutional Neural Networks (CNN) models across multiple devices. Our key insight is that breaking the memory bottleneck requires splitting inference at sub-layer granularity rather than at layer boundaries. We reinterpret pre-trained models to enable kernel-wise and neuron-wise partitioning, and distribute both model parameters and intermediate activations across multiple MCUs. A lightweight, resource-aware coordinator orchestrates the inference across MCU devices with heterogeneous resources. We implement the proposed system on a real testbed and evaluate it on up to 8 MCUs using MobileNetV2, a representative CNN model. Our experimental results show that CNN models infeasible on a single MCU can be executed across networked MCUs, reducing the per-MCU peak RAM usage while maintaining the practical end-to-end inference latency. All the source code of this work can be found here: https://github.com/shashsuresh/split-inference-on-MCUs.

Related papers

• NanoCockpit: Performance-optimized Application Framework for AI-based Autonomous Nanorobotics [50.594459728605734]
  Small form factor, i.e., a few 10s grams, severely limits onboard computational resources to sub-SI100milliwatt microcontroller units (MCUs)<n>Our framework achieves ideal end-to-end latency, i.e. zero overhead due to serialized tasks, delivering quantifiable improvements in closed-loop control performance.
  arXiv  Detail & Related papers  (2026-01-12T12:29:38Z)
• RockNet: Distributed Learning on Ultra-Low-Power Devices [49.01692357536576]
  This paper presents RockNet, a new TinyML method tailored for ultra-low-power hardware.<n>By leveraging that CPS consist of multiple devices, we design a distributed learning method that integrates Machine Learning and wireless communication.<n>Our results show that a tight integration of distributed ML, distributed computing, and communication enables, for the first time, training on ultra-low-power hardware with state-of-the-art accuracy.
  arXiv  Detail & Related papers  (2025-10-15T09:09:30Z)
• msf-CNN: Patch-based Multi-Stage Fusion with Convolutional Neural Networks for TinyML [0.20052993723676893]
  We introduce msf-CNN, a novel technique that efficiently finds optimal fusion settings for convolutional neural networks (CNNs)<n>We show that msf-CNN can achieve inference using 50% less RAM compared to the prior art.
  arXiv  Detail & Related papers  (2025-05-16T17:47:15Z)
• NNsight and NDIF: Democratizing Access to Open-Weight Foundation Model Internals [58.83169560132308]
  We introduce NNsight and NDIF, technologies that work in tandem to enable scientific study of the representations and computations learned by very large neural networks.
  arXiv  Detail & Related papers  (2024-07-18T17:59:01Z)
• MCU-MixQ: A HW/SW Co-optimized Mixed-precision Neural Network Design Framework for MCUs [9.719789698194154]
  Mixed-precision neural network (MPNN) that utilizes just enough data width for the neural network processing is an effective approach to meet the stringent resources constraints. However, there is still a lack of sub-byte and mixed-precision SIMD operations in MCU-class ISA. In this work, we propose to pack multiple low-bitwidth arithmetic operations within a single instruction multiple data (SIMD) instructions in typical MCUs.
  arXiv  Detail & Related papers  (2024-07-17T14:51:15Z)
• Basic Binary Convolution Unit for Binarized Image Restoration Network [146.0988597062618]
  In this study, we reconsider components in binary convolution, such as residual connection, BatchNorm, activation function, and structure, for image restoration tasks. Based on our findings and analyses, we design a simple yet efficient basic binary convolution unit (BBCU) Our BBCU significantly outperforms other BNNs and lightweight models, which shows that BBCU can serve as a basic unit for binarized IR networks.
  arXiv  Detail & Related papers  (2022-10-02T01:54:40Z)
• MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning [72.80896338009579]
  We find that the memory bottleneck is due to the imbalanced memory distribution in convolutional neural network (CNN) designs. We propose a generic patch-by-patch inference scheduling, which significantly cuts down the peak memory. We automate the process with neural architecture search to jointly optimize the neural architecture and inference scheduling, leading to MCUNetV2.
  arXiv  Detail & Related papers  (2021-10-28T17:58:45Z)
• Differentiable Network Pruning for Microcontrollers [14.864940447206871]
  We present a differentiable structured network pruning method for convolutional neural networks. It integrates a model's MCU-specific resource usage and parameter importance feedback to obtain highly compressed yet accurate classification models.
  arXiv  Detail & Related papers  (2021-10-15T20:26:15Z)
• MicroNets: Neural Network Architectures for Deploying TinyML Applications on Commodity Microcontrollers [18.662026553041937]
  Machine learning on resource constrained microcontrollers (MCUs) promises to drastically expand the application space of the Internet of Things (IoT) TinyML presents severe technical challenges, as deep neural network inference demands a large compute and memory budget. neural architecture search (NAS) promises to help design accurate ML models that meet the tight MCU memory, latency and energy constraints.
  arXiv  Detail & Related papers  (2020-10-21T19:39:39Z)
• Leveraging Automated Mixed-Low-Precision Quantization for tiny edge microcontrollers [76.30674794049293]
  This paper presents an automated mixed-precision quantization flow based on the HAQ framework but tailored for the memory and computational characteristics of MCU devices. Specifically, a Reinforcement Learning agent searches for the best uniform quantization levels, among 2, 4, 8 bits, of individual weight and activation tensors. Given an MCU-class memory bound to 2MB for weight-only quantization, the compressed models produced by the mixed-precision engine result as accurate as the state-of-the-art solutions.
  arXiv  Detail & Related papers  (2020-08-12T06:09:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.