Efficient Sensor Fusion for Gesture Recognition on Resource-Constrained Devices

• URL: http://arxiv.org/abs/2605.13462v1
• Date: Wed, 13 May 2026 12:53:22 GMT
• Title: Efficient Sensor Fusion for Gesture Recognition on Resource-Constrained Devices
• Abstract summary: Traditional vision-based approaches face challenges regarding power consumption, computational latency, and user privacy.<n>This paper proposes a lightweight, privacy-preserving gesture recognition system based on the fusion of low-resolution Time-of-Flight (ToF) and Infrared (IR) thermal sensors.<n>A compact Convolutional Neural Network (CNN) with a specialized grouped-convolution architecture is designed to fuse these modalities efficiently on a microcontroller.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Gesture recognition is a cornerstone of Human-Computer Interaction (HCI) for smart eyewear, enabling natural and device-free control in augmented reality environments. Traditional vision-based approaches face significant challenges regarding power consumption, computational latency, and user privacy. This paper proposes a lightweight, privacy-preserving gesture recognition system based on the fusion of low-resolution Time-of-Flight (ToF) and Infrared (IR) thermal sensors. We used an 8 times 8 multizone ToF sensor (VL53L8CH) and an 8 times 8 IR array (AMG8833) to capture complementary depth and thermal cues. A compact Convolutional Neural Network (CNN) with a specialized grouped-convolution architecture is designed to fuse these modalities efficiently on a microcontroller (MCU). Experimental results on a custom dataset of 7 static gestures, validated via k-fold cross-validation, demonstrate that the proposed fusion strategy significantly outperforms single-sensor baselines with an accuracy of 92.3% and a macro F1-score of 0.93. Finally, on-device benchmarks on STM32F4 and STM32H7 MCUs confirm the system's suitability for resource-constrained wearables, requiring only 6,343 parameters and achieving millisecond-level inference latency with a total system power of 50 mW.

Related papers

• Robust Slip Detection and Material Classification via Spatiotemporal Transformers on a Uniformly-Illuminated Visuo-Tactile Sensor [52.33204893988409]
  We propose a visuo-tactile sensor featuring customized uniform illumination, alongside a unified perception framework.<n>The proposed hardware-software framework provides high-fidelity feedback and a powerful multi-modal perception baseline for complex robotic manipulation.
  arXiv  Detail & Related papers  (2026-08-25T07:25:27Z)
• Seeing Through WiFi: Lightweight Human Pose Estimation with Dynamic Kernel Attention [55.570606479386434]
  WiFi-based human pose estimation (HPE) enables the detection and interpretation of human body positions and movements without the need for wearable devices.<n>This paper introduces a novel framework, WiLHPE, for lightweight and efficient human pose estimation using WiFi CSI signals.
  arXiv  Detail & Related papers  (2026-07-03T11:04:05Z)
• GenAI for Energy-Efficient and Interference-Aware Compressed Sensing of GNSS Signals on a Google Edge TPU [37.74014735747033]
  This paper introduces a novel method for compressing and classifying jamming threats using generative artificial intelligence (GenAI)<n>Our proposed approach efficiently compresses data streams directly at the hardware receiver while simultaneously classifying jamming and spoofing attacks in real time.<n>The hardware-centric GenAI approach also substantially reduces jammer signal transmission costs, offering a practical solution for interference mitigation.
  arXiv  Detail & Related papers  (2026-05-14T13:43:55Z)
• Design and Validation of a Lightweight 1D CNN for Affective Touch Classification in Soft Plush Companions [59.49967357689445]
  Soft, sensorized companions offer a physically safe and emotionally intuitive interface for socially assistive technologies.<n>This study presents a complete open-source dilated real-based framework for the development and validation of deep learning models for affective touch recognition in soft interactive companions.
  arXiv  Detail & Related papers  (2026-04-16T10:20:05Z)
• Development of ML model for triboelectric nanogenerator based sign language detection system [0.0]
  Sign language recognition is vital for bridging communication gaps between deaf and hearing communities.<n>This work presents a comparison of machine learning (ML) and deep learning models for a custom triboelectric nanogenerator (TENG)-based sensor glove.<n>The proposed MFCC CNN-LSTM architecture processes frequency-domain features from each sensor through independent convolutional branches before fusion.
  arXiv  Detail & Related papers  (2026-03-26T08:35:02Z)
• Real-Time Human Activity Recognition on Edge Microcontrollers: Dynamic Hierarchical Inference with Multi-Spectral Sensor Fusion [7.184610830886172]
  We propose a resource-aware hierarchical network based on multi-spectral fusion and interpretable modules.<n> Deployed on an ARM Cortex-M4 microcontroller for low-power real-time inference, HPPI-Net achieves 96.70% accuracy.<n>Compared with MobileNetV3, HPPI-Net improves accuracy by 1.22% and reduces RAM usage by 71.2% and ROM usage by 42.1%.
  arXiv  Detail & Related papers  (2026-01-29T15:21:45Z)
• Neuromorphic Eye Tracking for Low-Latency Pupil Detection [37.091037454305024]
  Eye tracking for wearable systems demands low latency and milliwatt-level power.<n>Neuromorphic sensors and spiking neural networks (SNNs) offer a promising alternative.<n>This paper presents a neuromorphic version of top-performing event-based eye-tracking models.
  arXiv  Detail & Related papers  (2025-12-10T11:30:21Z)
• STAR: A Privacy-Preserving, Energy-Efficient Edge AI Framework for Human Activity Recognition via Wi-Fi CSI in Mobile and Pervasive Computing Environments [0.0]
  Human Activity Recognition via Wi-Fi Channel State Information (CSI) presents a privacy-preserving, contactless sensing approach suitable for smart homes, healthcare monitoring, and mobile IoT systems.<n>This paper proposes STAR (Sensing Technology for Activity Recognition), an edge-AI-optimized framework that integrates a lightweight neural architecture, adaptive signal processing, and hardware-aware co-optimization.<n>With sub-second response latency and low power consumption, the system ensures real-time, privacy-preserving HAR, offering a practical, scalable solution for mobile and pervasive computing environments.
  arXiv  Detail & Related papers  (2025-10-30T05:08:25Z)
• End-to-End Efficiency in Keyword Spotting: A System-Level Approach for Embedded Microcontrollers [0.18472148461613155]
  Keywords spotting (KWS) is a key enabling technology for hands-free interaction in embedded and IoT devices, where stringent memory and energy constraints challenge the deployment of AI-enabeld devices.<n>In this work, we evaluate and compare several state-of-the-art lightweight neural network architectures, including DS-CNN, LiCoNet, and TENet, alongside our proposed Typman-KWS architecture built upon MobileNet, specifically designed for efficient KWS on microcontroller units (MCUs)<n>Our results show that TKWS with three residual blocks achieves up to 92.4% F1-score with only 14.4k parameters
  arXiv  Detail & Related papers  (2025-09-08T16:01:55Z)
• SpikeSMOKE: Spiking Neural Networks for Monocular 3D Object Detection with Cross-Scale Gated Coding [40.87826765639894]
  SpikeSMOKE is a new attempt for low-power monocular 3D object detection.<n>SpikeSMOKE can significantly reduce energy consumption compared to the results on SMOKE.
  arXiv  Detail & Related papers  (2025-06-09T13:20:10Z)
• FemtoDet: An Object Detection Baseline for Energy Versus Performance Tradeoffs [27.006082622843653]
  Vision applications of convolutional neural networks, such as always-on surveillance cameras, are critical for energy constraints. This paper aims to serve as a baseline by designing detectors to reach tradeoffs between energy and performance from two perspectives.
  arXiv  Detail & Related papers  (2023-01-17T06:24:08Z)
• Energy-efficient and Privacy-aware Social Distance Monitoring with Low-resolution Infrared Sensors and Adaptive Inference [4.158182639870093]
  Low-resolution infrared (IR) sensors can be leveraged to implement privacy-preserving social distance monitoring solutions in indoor spaces. We propose an energy-efficient adaptive inference solution consisting of a cascade of a simple wake-up trigger and a 8-bit quantized Convolutional Neural Network (CNN) We show that, when processing the output of a 8x8 low-resolution IR sensor, we are able to reduce the energy consumption by 37-57% with respect to a static CNN-based approach.
  arXiv  Detail & Related papers  (2022-04-22T07:07:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.