TinySSL: Distilled Self-Supervised Pretraining for Sub-Megabyte MCU Models

• URL: http://arxiv.org/abs/2605.08241v1
• Date: Thu, 07 May 2026 07:51:00 GMT
• Title: TinySSL: Distilled Self-Supervised Pretraining for Sub-Megabyte MCU Models
• Abstract summary: Self-supervised learning (SSL) has transformed representation learning for large models, yet remains unexplored for microcontroller (MCU)-class models with fewer than 500K parameters.<n>We propose Capacity-Aware Distilled Self-Supervised Learning (CA-DSSL), a teacher-guided framework that overcomes them without labels or text supervision.<n>Preliminary ImageNet-100 experiments reveal that CA-DSSL's advantage is specific to small-data regimes.
• Score: 0.14504054468850666
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Self-supervised learning (SSL) has transformed representation learning for large models, yet remains unexplored for microcontroller (MCU)-class models with fewer than 500K parameters. We identify three obstacles at this scale -- projection head dominance, representation bottleneck, and augmentation sensitivity -- and propose Capacity-Aware Distilled Self-Supervised Learning (CA-DSSL), a teacher-guided framework that overcomes them without labels or text supervision. CA-DSSL combines asymmetric distillation from a frozen DINO ViT-S/16 teacher, multi-scale feature distillation for spatial representations, and a progressive augmentation curriculum. On a MobileNetV2-0.35 backbone (396K parameters) pretrained on CIFAR-100, CA-DSSL reaches 62.7 0.5% linear-probe accuracy (3-seed mean) -- surpassing SimCLR-Tiny by 18 pp, matching SEED (61.7%) with 10 fewer projection parameters (426K vs. 3.15M), and reaching 94.0% of a supervised upper bound. Standard SSL methods (BYOL-Tiny, DINO-Tiny) collapse entirely at this scale. On Pascal VOC detection, CA-DSSL achieves 2.3 the mAP of random initialization and +3 pp over SEED, though SimCLR-Tiny matches CA-DSSL on detection mAP. The deployed backbone occupies 378 KB (INT8) with no inference overhead from pretraining. Preliminary ImageNet-100 experiments reveal that CA-DSSL's advantage is specific to small-data regimes; scaling to ImageNet-1K is discussed as future work.

Related papers

• Cross-Architecture Knowledge Distillation from a Vision Foundation Model to a Lightweight Visual State Space Model for Tea Leaf Disease Classification [10.122947222129108]
  Self-supervised vision foundation models such as DINOv2 provide strong features but are too large for field deployment.<n>We study cross-architecture knowledge distillation from a fine-tuned DINOv2 teacher to a compact bidirectional Visual State Space Model.<n>We identify and fix two training-stability problems that prevent the from-scratch SSM student from learning on limited data.
  arXiv  Detail & Related papers  (2026-08-27T08:02:05Z)
• Towards Robust Reinforcement Learning for Small-Scale Language Model Agents [5.301609879131692]
  In this paper, fifteen (model, corpus) configurations were trained using Proximal Policy Optimization (PPO)<n>The experiments included Pythia-70M, 160M, 410M and SmolLM2-135M, 360M on the TinyStories, CNN/DailyMail, and Wiki-text103 corpora.<n>The proposed system converged stably in all experiments and improved preference win rate over the SFT baseline in configurations with a fluent prior and an informative reward signal.
  arXiv  Detail & Related papers  (2026-07-27T21:30:44Z)
• Traffic-Aware Randomized Smoothing for LLM-Based Network Intrusion Detection [1.4915002678801434]
  We present Traffic-Aware Randomized Smoothing (TA-RS) for intrusion detection systems.<n>It is a certified defense that injects Gaussian noise exclusively into the controllable (DC) subspace.<n>It achieves 55-100% certified accuracy on two datasets.
  arXiv  Detail & Related papers  (2026-07-15T13:08:19Z)
• Learning to Solve Generative ODEs Beyond the Linear Span [50.13853710831612]
  We propose SpanLift, a lightweight neural solver that augments scalar-coefficient updates with a spatial residual operator.<n>Across pixel-space diffusion, latent flow matching, and precipitation nowcasting, SpanLift achieves state-of-the-art few-step sampling.
  arXiv  Detail & Related papers  (2026-06-07T15:22:13Z)
• Trust-SSL: Additive-Residual Selective Invariance for Robust Aerial Self-Supervised Learning [6.479513353756828]
  Self-supervised learning (SSL) is a standard approach for representation learning in aerial imagery.<n>This study proposes a training strategy and architectural modification to enhance SSL to such corruptions.<n>The method achieves the highest mean linear-probe accuracy among six backbones on EuroSAT, AID, and NWPU-RESISC45.
  arXiv  Detail & Related papers  (2026-04-23T07:07:59Z)
• DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning [61.10299147201369]
  This paper introduces a novel autonomous RL approach, called DigiRL, for training in-the-wild device control agents. We build a scalable and parallelizable Android learning environment equipped with a VLM-based evaluator. We demonstrate the effectiveness of DigiRL using the Android-in-the-Wild dataset, where our 1.3B VLM trained with RL achieves a 49.5% absolute improvement.
  arXiv  Detail & Related papers  (2024-06-14T17:49:55Z)
• TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritance [97.01406871579525]
  We propose a novel cross-modal distillation method, called TinyCLIP, for large-scale language-image pre-trained models. We show that TinyCLIP can reduce the size of the pre-trained CLIP ViT-B/32 by 50%, while maintaining comparable zero-shot performance. Our TinyCLIP ViT-8M/16, trained on YFCC-15M, achieves an impressive zero-shot top-1 accuracy of 41.1% on ImageNet.
  arXiv  Detail & Related papers  (2023-09-21T17:59:53Z)
• Speech separation with large-scale self-supervised learning [41.96634125460265]
  Self-supervised learning (SSL) methods such as WavLM have shown promising speech separation (SS) results in small-scale simulation-based experiments. We extend the exploration of the SSL-based SS by massively scaling up both the pre-training data (more than 300K hours) and fine-tuning data (10K hours)
  arXiv  Detail & Related papers  (2022-11-09T20:00:21Z)
• Elastic Weight Consolidation Improves the Robustness of Self-Supervised Learning Methods under Transfer [4.2141621237414615]
  Self-supervised representation learning (SSL) methods provide an effective label-free initial condition for fine-tuning downstream tasks. We re-interpret SSL fine-tuning under the lens of Bayesian continual learning and consider regularization through the Elastic Weight Consolidation (EWC) framework. We demonstrate that self-regularization against an initial SSL backbone improves worst sub-group performance in Waterbirds by 5% and Celeb-A by 2%.
  arXiv  Detail & Related papers  (2022-10-28T19:00:25Z)
• Alexa Teacher Model: Pretraining and Distilling Multi-Billion-Parameter Encoders for Natural Language Understanding Systems [63.713297451300086]
  We present results from a large-scale experiment on pretraining encoders with non-embedding parameter counts ranging from 700M to 9.3B. Their subsequent distillation into smaller models ranging from 17M-170M parameters, and their application to the Natural Language Understanding (NLU) component of a virtual assistant system.
  arXiv  Detail & Related papers  (2022-06-15T20:44:23Z)
• DATA: Domain-Aware and Task-Aware Pre-training [94.62676913928831]
  We present DATA, a simple yet effective NAS approach specialized for self-supervised learning (SSL) Our method achieves promising results across a wide range of computation costs on downstream tasks, including image classification, object detection and semantic segmentation.
  arXiv  Detail & Related papers  (2022-03-17T02:38:49Z)
• Measuring Self-Supervised Representation Quality for Downstream Classification using Discriminative Features [56.89813105411331]
  We study the representation space of state-of-the-art self-supervised models including SimCLR, SwaV, MoCo, BYOL, DINO, SimSiam, VICReg and Barlow Twins. We propose Self-Supervised Representation Quality Score (or Q-Score), an unsupervised score that can reliably predict if a given sample is likely to be mis-classified. Fine-tuning with Q-Score regularization can boost the linear probing accuracy of SSL models by up to 5.8% on ImageNet-100 and 3.7% on ImageNet-1K.
  arXiv  Detail & Related papers  (2022-03-03T17:48:23Z)
• Exploiting Large-scale Teacher-Student Training for On-device Acoustic Models [15.237992590162593]
  We present results from Alexa speech teams on semi-supervised learning (SSL) of acoustic models (AM) We discuss SSL for AMs in a small footprint setting, showing that a smaller capacity model trained with 1 million hours of unsupervised data can outperform a baseline supervised system by 14.3% word error rate reduction (WERR) We then switch to SSL using larger student models in low data regimes; while learning efficiency with unsupervised data is higher, student models may outperform teacher models in such a setting.
  arXiv  Detail & Related papers  (2021-06-11T02:23:40Z)
• A Simple Semi-Supervised Learning Framework for Object Detection [55.95789931533665]
  Semi-supervised learning (SSL) has a potential to improve the predictive performance of machine learning models using unlabeled data. We propose STAC, a simple yet effective SSL framework for visual object detection along with a data augmentation strategy.
  arXiv  Detail & Related papers  (2020-05-10T19:15:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.