OpenGlass: Ultra-Low-Power On-Device AI Eyewear with Event-based Vision

• URL: http://arxiv.org/abs/2606.07431v2
• Date: Mon, 08 Jun 2026 14:50:28 GMT
• Title: OpenGlass: Ultra-Low-Power On-Device AI Eyewear with Event-based Vision
• Abstract summary: This work introduces an open-source smart glasses platform for rapid prototyping of novel sensors and algorithms.<n>Its modular design uses a flexible FPC interposer to support both event-based and frame-based cameras without full PCB.<n>The prototype achieves up to 11.5 hours of continuous on-device ML from a 200 mAh battery.
• Score: 8.93031966791759
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Smart eyewear enables unobtrusive, context-aware interaction through multimodal sensors and on-device intelligence, but is severely limited by power, memory, and compute constraints in a compact form factor. Open-hardware platforms supporting event-based vision and embedded ML at this scale are rare. This work introduces an open-source smart glasses platform for rapid prototyping of novel sensors and algorithms. Its modular design uses a flexible FPC interposer to support both event-based and frame-based cameras without full PCB redesign. A hardware-software co-designed power management system combines a configurable PMIC with event-driven wake-up via an nRF5340 coordinator, keeping the GAP9 RISC-V SoC powered down between inferences. The prototype achieves up to 11.5 hours of continuous on-device ML from a 200 mAh battery. As a demonstration, an egocentric hand gesture recognition pipeline was evaluated on the LynX dataset using polarity-separated event histograms from a Prophesee GENX320 camera. R(2+1)D achieved the best cross-subject accuracy of 83.94\% (macro F1 = 0.781) under leave-two-subjects-out validation, with 78.3 ms end-to-end inference latency on the GAP9. Temporal augmentation and removal of ambiguous classes provided the largest gains (+8.9 pp). All hardware designs, firmware, and models are released open source.

Related papers

• Low-Power License Plate Detection and Recognition on a RISC-V Multi-Core MCU-Based Vision System [15.675111358084235]
  We present the first (to the best of our knowledge) demonstration of a low-power MCU-based edge device for Automatic License Plate Recognition (ALPR)<n>The design leverages on a 9-core RISC-V processor, GAP8, coupled with a QVGA ultra-low-power greyscale imager.<n>The proposed visual processing pipeline uses a multi-model inference approach based on SSDlite-MobilenetV2 for license plate detection and LPRNet for optical character recognition, reaching a 38.9% mAP score for the first task and a recognition rate of >99.13% for the latter on public datasets.
  arXiv  Detail & Related papers  (2026-07-07T09:24:44Z)
• Fully-sensorized smart-eyewear platform for on-device Machine Learning [1.3651611781068242]
  ARGO is a smart eyewear platform designed to bridge ergonomic comfort, high computational throughput, and energy efficiency.<n>The primary contribution lies in the holistic co-design of hardware, firmware, and artificial intelligence.<n>The platform integrates a multimodal sensor suite, RGB cameras, Time-of-Flight sensors, microphones, and ambient sensors, and delivers 10 FPS at a continuous autonomy of 113 minutes on a 200 mAh battery.
  arXiv  Detail & Related papers  (2026-06-17T17:11:39Z)
• Embedded Arena: Iterative Optimization via Hardware Feedback [44.54626665766497]
  Agentic co-optimization achieves 250x compression for vision models with 3.3% accuracy loss and 400x for audio with 6% Feature Error Rate loss.<n>We demonstrate practical impact in two real-world systems: an elk-detection camera trap (96.7% accuracy) and a phonetic-transcription wearable (8.44% FER)
  arXiv  Detail & Related papers  (2026-06-15T04:04:23Z)
• Tiny Collaborative Inference for Occlusion-Robust Object Detection [0.5352699766206808]
  Edge AI nodes for search and rescue are increasingly expected to run computer vision locally.<n>Ultra-low-end hardware imposes hard constraints on memory, compute, and inter-device communication.<n>This work addresses occlusion-robust object detection devices with less than 1 MB by combining an MCU backbone, a YOLOv2 detection head, and Lite quantisation.
  arXiv  Detail & Related papers  (2026-06-01T21:10:08Z)
• Dexora: Open-source VLA for High-DoF Bimanual Dexterity [97.98653150757526]
  We introduce Dexora, the first open-source VLA system that targets dual-arm, dual-hand high-DoF manipulation.<n>We design a hybrid teleoperation pipeline that decouples gross arm kinematics from fine finger motion.<n>We show that Dexora outperforms competitive VLA baselines on both basic and dexterous benchmarks.
  arXiv  Detail & Related papers  (2026-05-18T17:50:32Z)
• Efficient Sensor Fusion for Gesture Recognition on Resource-Constrained Devices [0.0]
  Traditional vision-based approaches face challenges regarding power consumption, computational latency, and user privacy.<n>This paper proposes a lightweight, privacy-preserving gesture recognition system based on the fusion of low-resolution Time-of-Flight (ToF) and Infrared (IR) thermal sensors.<n>A compact Convolutional Neural Network (CNN) with a specialized grouped-convolution architecture is designed to fuse these modalities efficiently on a microcontroller.
  arXiv  Detail & Related papers  (2026-05-13T12:53:22Z)
• STEP3-VL-10B Technical Report [115.89015065130127]
  STEP3-VL-10B is a lightweight foundation model designed to redefine the trade-off between compact efficiency and frontier-level multimodal intelligence.<n>We implement Parallel Coordinated Reasoning (PaCoRe) to scale test-time compute, allocating resources to scalable perceptual reasoning.<n>It records 92.2% on MMBench and 80.11% on MMMU, while excelling in complex reasoning with 94.43% on AIME2025 and 75.95% on MathVision.
  arXiv  Detail & Related papers  (2026-01-14T17:58:24Z)
• NanoCockpit: Performance-optimized Application Framework for AI-based Autonomous Nanorobotics [50.594459728605734]
  Small form factor, i.e., a few 10s grams, severely limits onboard computational resources to sub-SI100milliwatt microcontroller units (MCUs)<n>Our framework achieves ideal end-to-end latency, i.e. zero overhead due to serialized tasks, delivering quantifiable improvements in closed-loop control performance.
  arXiv  Detail & Related papers  (2026-01-12T12:29:38Z)
• RockNet: Distributed Learning on Ultra-Low-Power Devices [49.01692357536576]
  This paper presents RockNet, a new TinyML method tailored for ultra-low-power hardware.<n>By leveraging that CPS consist of multiple devices, we design a distributed learning method that integrates Machine Learning and wireless communication.<n>Our results show that a tight integration of distributed ML, distributed computing, and communication enables, for the first time, training on ultra-low-power hardware with state-of-the-art accuracy.
  arXiv  Detail & Related papers  (2025-10-15T09:09:30Z)
• StrikeWatch: Wrist-worn Gait Recognition with Compact Time-series Models on Low-power FPGAs [10.946464973530214]
  Proper gait patterns can lead to injuries, particularly without expert feedback.<n>Wrist-worn wearables offer a more practical and non-intrusive alternative.<n>This paper introduces StrikeWatch, a compact wrist-worn system that performs entirely on-device, real-time gait recognition.
  arXiv  Detail & Related papers  (2025-10-14T20:28:31Z)
• Tiny but Mighty: A Software-Hardware Co-Design Approach for Efficient Multimodal Inference on Battery-Powered Small Devices [12.40080721078945]
  Large Multimodal Models (LMMs) are inherently modular, consisting of vision and audio encoders, projectors, and large language models.<n>Yet, they are almost always executed monolithically, which underutilizes the heterogeneous accelerators.<n>We present a hardware-software co-design inference framework for Large Multimodal Models (LMMs)<n>This framework breaks large models into modular components and scheduled to run on the most appropriate compute units.
  arXiv  Detail & Related papers  (2025-09-25T22:28:44Z)
• A Heterogeneous RISC-V based SoC for Secure Nano-UAV Navigation [40.8381466360025]
  nano-UAVs face significant power and payload constraints while requiring advanced computing capabilities. We present Shaheen, a 9mm2 200mW system-on-a-chip (SoC) It integrates a Linux-capable RV64 core, compliant with the v1.0 ratified Hypervisor extension, along with a low-cost and low-power memory controller. At the same time, it integrates a fully programmable energy- and area-efficient multi-core cluster of RV32 cores optimized for general-purpose DSP.
  arXiv  Detail & Related papers  (2024-01-07T16:03:47Z)
• FastPillars: A Deployment-friendly Pillar-based 3D Detector [63.0697065653061]
  Existing BEV-based (i.e., Bird Eye View) detectors favor sparse convolutions (known as SPConv) to speed up training and inference. FastPillars delivers state-of-the-art accuracy on Open dataset with 1.8X speed up and 3.8 mAPH/L2 improvement over CenterPoint (SPConv-based)
  arXiv  Detail & Related papers  (2023-02-05T12:13:27Z)
• FastFlowNet: A Lightweight Network for Fast Optical Flow Estimation [81.76975488010213]
  Dense optical flow estimation plays a key role in many robotic vision tasks. Current networks often occupy large number of parameters and require heavy computation costs. Our proposed FastFlowNet works in the well-known coarse-to-fine manner with following innovations.
  arXiv  Detail & Related papers  (2021-03-08T03:09:37Z)
• ASFD: Automatic and Scalable Face Detector [129.82350993748258]
  We propose a novel Automatic and Scalable Face Detector (ASFD) ASFD is based on a combination of neural architecture search techniques as well as a new loss design. Our ASFD-D6 outperforms the prior strong competitors, and our lightweight ASFD-D0 runs at more than 120 FPS with Mobilenet for VGA-resolution images.
  arXiv  Detail & Related papers  (2020-03-25T06:00:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.