TinyVLM: Zero-Shot Object Detection on Microcontrollers via Vision-Language Distillation with Matryoshka Embeddings

• URL: http://arxiv.org/abs/2603.00136v1
• Date: Tue, 24 Feb 2026 05:32:51 GMT
• Title: TinyVLM: Zero-Shot Object Detection on Microcontrollers via Vision-Language Distillation with Matryoshka Embeddings
• Abstract summary: We present TinyVLM, the first framework enabling zero-shot object detection on resource-constrained MCUs with less than 1MB of memory.<n>TinyVLM achieves competitive zero-shot accuracy on COCO, Flowers102, and Food101 while requiring only 285KB of RAM and 892KB of flash memory.
• Score: 0.14504054468850666
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Zero-shot object detection enables recognising novel objects without task-specific training, but current approaches rely on large vision language models (VLMs) like CLIP that require hundreds of megabytes of memory - far exceeding the constraints of micro controller units (MCUs). We present TinyVLM, the first framework enabling zero-shot object detection on resource-constrained MCUs with less than 1MB of memory. Our approach introduces three key innovations: (1) a decoupled architecture that separates visual inference from text encoding, allowing precomputed class embeddings to be stored in flash memory; (2) Matryoshka distillation that trains nested embeddings at multiple dimensions (16-256), enabling flexible accuracy-memory trade-offs; and (3) quantized embedding storage that reduces class prototype memory by 4x with minimal accuracy loss. Trained on Conceptual Captions 3M (CC3M), TinyVLM achieves competitive zero-shot accuracy on COCO, Flowers102, and Food101 while requiring only 285KB of RAM and 892KB of flash memory for the deployed vision encoder. We demonstrate real-time inference at 26 FPS on STM32H7 and over 1,000 FPS on MAX78000 with its CNN accelerator, enabling practical zero-shot detection on edge devices for the first time.

Related papers

• MiLSD: A Micro Line-Segment Detector for Resource-Constrained Devices [1.098959303541186]
  Line segment detection is a key building block in visual SLAM, 3D reconstruction, and industrial inspection.<n>Recent deep learning methods have greatly improved accuracy, yet even the smallest models require several megabytes of memory.<n>We propose MiLSD, a detector tailored for MCU-level constraints.
  arXiv  Detail & Related papers  (2026-07-07T00:04:50Z)
• Latent Replay Detection: Memory-Efficient Continual Object Detection on Microcontrollers via Task-Adaptive Compression [0.14504054468850666]
  Existing continual learning methods require storing raw images far exceeding MCU memory budgets.<n>We present Latent Replay Detection (LRD), the first framework for continual object detection under MCU memory constraints.<n> Deployed on STM32H753ZI, ESP32-S3, and MAX78000 MCUs, LRD achieves 4.9-97.5ms latency per inference within a 64KB memory budget.
  arXiv  Detail & Related papers  (2026-02-24T07:35:27Z)
• FlashSVD: Memory-Efficient Inference with Streaming for Low-Rank Models [15.244129138320782]
  FlashSVD is an end-to-end rank-aware streaming inference framework for SVD-compressed large language models.<n>It cuts peak activation memory by up to 70.2% and intermediate transient memory by 75%.<n>It incurs no accuracy loss with upstream encodering compression methods, offering a practical path toward memory-constrained deployment of low-rank LLMs.
  arXiv  Detail & Related papers  (2025-08-02T22:06:46Z)
• Designing Extremely Memory-Efficient CNNs for On-device Vision Tasks [2.9835839258066015]
  We introduce a memory-efficient CNN (convolutional neural network) for on-device vision tasks. The proposed network classifies ImageNet with extremely low memory (i.e., 63 KB) while achieving competitive top-1 accuracy (i.e., 61.58%) To the best of our knowledge, the memory usage of the proposed network is far smaller than state-of-the-art memory-efficient networks.
  arXiv  Detail & Related papers  (2024-08-07T10:04:04Z)
• vTensor: Flexible Virtual Tensor Management for Efficient LLM Serving [53.972175896814505]
  Large Language Models (LLMs) are widely used across various domains, processing millions of daily requests. Large Language Models (LLMs) are widely used across various domains, processing millions of daily requests.
  arXiv  Detail & Related papers  (2024-07-22T14:37:58Z)
• MCUFormer: Deploying Vision Transformers on Microcontrollers with Limited Memory [76.02294791513552]
  We propose a hardware-algorithm co-optimizations method called MCUFormer to deploy vision transformers on microcontrollers with extremely limited memory. Experimental results demonstrate that our MCUFormer achieves 73.62% top-1 accuracy on ImageNet for image classification with 320KB memory.
  arXiv  Detail & Related papers  (2023-10-25T18:00:26Z)
• TinyissimoYOLO: A Quantized, Low-Memory Footprint, TinyML Object Detection Network for Low Power Microcontrollers [1.2846479438896337]
  This paper introduces a highly flexible, quantized, memory-efficient, and ultra-lightweight object detection network, called TinyissimoYOLO. It aims to enable object detection on microcontrollers in the power domain of milliwatts, with less than 0.5MB memory available for storing convolutional neural network weights.
  arXiv  Detail & Related papers  (2023-05-22T12:57:38Z)
• MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning [72.80896338009579]
  We find that the memory bottleneck is due to the imbalanced memory distribution in convolutional neural network (CNN) designs. We propose a generic patch-by-patch inference scheduling, which significantly cuts down the peak memory. We automate the process with neural architecture search to jointly optimize the neural architecture and inference scheduling, leading to MCUNetV2.
  arXiv  Detail & Related papers  (2021-10-28T17:58:45Z)
• A TinyML Platform for On-Device Continual Learning with Quantized Latent Replays [66.62377866022221]
  Latent Replay-based Continual Learning (CL) techniques enable online, serverless adaptation in principle. We introduce a HW/SW platform for end-to-end CL based on a 10-core FP32-enabled parallel ultra-low-power processor. Our results show that by combining these techniques, continual learning can be achieved in practice using less than 64MB of memory.
  arXiv  Detail & Related papers  (2021-10-20T11:01:23Z)
• Leveraging Automated Mixed-Low-Precision Quantization for tiny edge microcontrollers [76.30674794049293]
  This paper presents an automated mixed-precision quantization flow based on the HAQ framework but tailored for the memory and computational characteristics of MCU devices. Specifically, a Reinforcement Learning agent searches for the best uniform quantization levels, among 2, 4, 8 bits, of individual weight and activation tensors. Given an MCU-class memory bound to 2MB for weight-only quantization, the compressed models produced by the mixed-precision engine result as accurate as the state-of-the-art solutions.
  arXiv  Detail & Related papers  (2020-08-12T06:09:58Z)
• TinyTL: Reduce Activations, Not Trainable Parameters for Efficient On-Device Learning [78.80707950262214]
  On-device learning enables edge devices to continually adapt the AI models to new data. Existing work solves this problem by reducing the number of trainable parameters. We present Tiny-Transfer-Learning (TinyTL) for memory-efficient on-device learning.
  arXiv  Detail & Related papers  (2020-07-22T18:39:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.