WebSerial Vision Training for Microcontrollers: A Browser-Based Companion to On-Device CNN Training

• URL: http://arxiv.org/abs/2604.22834v1
• Date: Mon, 20 Apr 2026 05:25:41 GMT
• Title: WebSerial Vision Training for Microcontrollers: A Browser-Based Companion to On-Device CNN Training
• Abstract summary: webmcu-vision-web is a browser-based application for end-to-end TinyML vision model training and deployment on the Seeed Studio XIAO ESP32-S3 Sense.<n>It provides a private, fully local machine learning pipeline, from flashing through image, CNN training, weight export, and live activation visualization.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper presents webmcu-vision-web, a single-file, zero-install browser application for end-to-end TinyML vision model training and deployment on the Seeed Studio XIAO ESP32-S3 Sense (XIAO ML Kit, $15--40 USD). Acting as a browser-based companion to the on-device Arduino firmware of Paper 1 [1], it provides a private, fully local machine learning pipeline, from firmware flashing through image collection, CNN training, weight export, and live activation visualization, without any software installation beyond a Chromium-based browser. The system targets educators, small businesses, and researchers who need to train task-specific visual classifiers under their exact deployment conditions. Key capabilities include: in-browser firmware flashing via esptool-js; an SD card file browser with image preview and inline editing; config.json live-sync for zero-recompile hyperparameter adjustment; webcam and ESP32 OV2640 camera image capture; TensorFlow.js CNN training completing a three-class run (~30 images per class, 20 epochs) in approximately 1 minute browser-side versus 9 minutes on-device, enabling a complete collect-train-deploy cycle in under 10 minutes; weight export as myWeights.bin and myWeights.h; confusion matrix; and a live Conv2 activation heatmap streamed from the ESP32 during inference. No data leaves the local machine at any stage. A five-run consistency evaluation on the three-class reference problem (0Blank, 1Cup, 2Pen) demonstrates stable convergence with mean accuracy and standard deviation reported; all artefacts are released at the repository link below. The repository is a living template for LLM-assisted adaptation to new hardware and tasks. All source code is MIT-licensed at https://github.com/webmcu-ai/webmcu-vision-web.

Related papers

• OpenLanguageModel: Readable and Composable Small-Language-Model Pretraining for Education and Research [1.1654568410166084]
  OpenLanguageModel (OLM) is an open-source PyTorch library for building and pretraining small language models.<n>The package includes 27 presets across nine familiar model families and documentation that progresses from LM fundamentals to architecture research.
  arXiv  Detail & Related papers  (2026-07-18T06:55:18Z)
• Stream3D-VLM: Online 3D Spatial Understanding with Incremental Geometry Priors [54.51965288351254]
  We present an online 3D vision-language model that enables real-time spatial understanding from streaming video.<n>Our approach adopts an autoregressive streaming control modeling based on the LLM's next-token prediction to learn when to respond.<n>To address the scarcity of streaming 3D-language data, we further develop a scalable data generation pipeline.
  arXiv  Detail & Related papers  (2026-06-05T04:16:24Z)
• On-Device Vision Training, Deployment, and Inference on a Thumb-Sized Microcontroller [0.0]
  This paper presents an end-to-end on-device vision machine learning pipeline, executing entirely on a microcontroller-class device costing $15-40 USD.<n>Running on the Seeed Studio ESP32-S3 XI Kit MLAO (8 MB PSRAM), the firmware achieves three-class 64x64 image classification in approximately 9 minutes per training run, with real-time inference at 6.3 FPS.
  arXiv  Detail & Related papers  (2026-04-24T20:59:16Z)
• A ROS 2 Wrapper for Florence-2: Multi-Mode Local Vision-Language Inference for Robotic Systems [0.0]
  This article presents a ROS 2 wrapper for Florence-2 that exposes the model through three complementary interaction modes.<n>The wrapper is designed for local execution and supports both native installation and Docker container deployment.
  arXiv  Detail & Related papers  (2026-04-01T17:29:59Z)
• Towards Universal Modal Tracking with Online Dense Temporal Token Learning [66.83607018706519]
  We propose a universal video-level modality-awareness tracking model with online dense temporal token learning.<n>We expand the model's inputs to a video sequence level, aiming to see a richer video context from a near-global perspective.
  arXiv  Detail & Related papers  (2025-07-27T08:47:42Z)
• Comgra: A Tool for Analyzing and Debugging Neural Networks [35.89730807984949]
  We introduce comgra, an open source python library for use with PyTorch. Comgra extracts data about the internal activations of a model and organizes it in a GUI. It can show both summary statistics and individual data points, compare early and late stages of training, focus on individual samples of interest, and visualize the flow of the gradient through the network.
  arXiv  Detail & Related papers  (2024-07-31T14:57:23Z)
• DistML.js: Installation-free Distributed Deep Learning Framework for Web Browsers [40.48978035180545]
  "DistML.js" is a library designed for training and inference of machine learning models within web browsers. We provide a comprehensive explanation of DistML.js's design, API, and implementation, alongside practical applications.
  arXiv  Detail & Related papers  (2024-07-01T07:13:14Z)
• Follow Anything: Open-set detection, tracking, and following in real-time [89.83421771766682]
  We present a robotic system to detect, track, and follow any object in real-time. Our approach, dubbed follow anything'' (FAn), is an open-vocabulary and multimodal model. FAn can be deployed on a laptop with a lightweight (6-8 GB) graphics card, achieving a throughput of 6-20 frames per second.
  arXiv  Detail & Related papers  (2023-08-10T17:57:06Z)
• Internet Explorer: Targeted Representation Learning on the Open Web [121.02587846761627]
  Modern vision models typically rely on fine-tuning general-purpose models pre-trained on large, static datasets. We propose dynamically utilizing the Internet to quickly train a small-scale model that does extremely well on the task at hand. Our approach, called Internet Explorer, explores the web in a self-supervised manner to progressively find relevant examples that improve performance on a desired target dataset.
  arXiv  Detail & Related papers  (2023-02-27T18:59:55Z)
• USB: A Unified Semi-supervised Learning Benchmark [125.25384569880525]
  Semi-supervised learning (SSL) improves model generalization by leveraging massive unlabeled data to augment limited labeled samples. Previous work typically trains deep neural networks from scratch, which is time-consuming and environmentally unfriendly. We construct a Unified SSL Benchmark (USB) by selecting 15 diverse, challenging, and comprehensive tasks.
  arXiv  Detail & Related papers  (2022-08-12T15:45:48Z)
• PyTorchVideo: A Deep Learning Library for Video Understanding [71.89124881732015]
  PyTorchVideo is an open-source deep-learning library for video understanding tasks. It covers a full stack of video understanding tools including multimodal data loading, transformations, and models. The library is based on PyTorch and can be used by any training framework.
  arXiv  Detail & Related papers  (2021-11-18T18:59:58Z)
• Rethinking Zero-shot Video Classification: End-to-end Training for Realistic Applications [26.955001807330497]
  Zero-shot learning (ZSL) trains a model once and generalizes to new tasks whose classes are not present in the training dataset. We propose the first end-to-end algorithm for ZSL in video classification. Our training procedure builds on insights from recent video classification literature and uses a trainable 3D CNN to learn the visual features.
  arXiv  Detail & Related papers  (2020-03-03T11:09:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.