Hand gesture recognition using 802.11ad mmWave sensor in the mobile device

• URL: http://arxiv.org/abs/2211.07090v1
• Date: Mon, 14 Nov 2022 03:36:17 GMT
• Title: Hand gesture recognition using 802.11ad mmWave sensor in the mobile device
• Authors: Yuwei Ren, Jiuyuan Lu, Andrian Beletchi, Yin Huang, Ilia Karmanov, Daniel Fontijne, Chirag Patel and Hao Xu
• Abstract summary: We explore the feasibility of AI assisted hand-gesture recognition using 802.11ad 60GHz (mmWave) technology in smartphones. We built a prototype system, where radar sensing and communication waveform can coexist by time-division duplex (TDD) It can gather sensing data and predict gestures within 100 milliseconds.
• Score: 2.5476515662939563
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We explore the feasibility of AI assisted hand-gesture recognition using 802.11ad 60GHz (mmWave) technology in smartphones. Range-Doppler information (RDI) is obtained by using pulse Doppler radar for gesture recognition. We built a prototype system, where radar sensing and WLAN communication waveform can coexist by time-division duplex (TDD), to demonstrate the real-time hand-gesture inference. It can gather sensing data and predict gestures within 100 milliseconds. First, we build the pipeline for the real-time feature processing, which is robust to occasional frame drops in the data stream. RDI sequence restoration is implemented to handle the frame dropping in the continuous data stream, and also applied to data augmentation. Second, different gestures RDI are analyzed, where finger and hand motions can clearly show distinctive features. Third, five typical gestures (swipe, palm-holding, pull-push, finger-sliding and noise) are experimented with, and a classification framework is explored to segment the different gestures in the continuous gesture sequence with arbitrary inputs. We evaluate our architecture on a large multi-person dataset and report > 95% accuracy with one CNN + LSTM model. Further, a pure CNN model is developed to fit to on-device implementation, which minimizes the inference latency, power consumption and computation cost. And the accuracy of this CNN model is more than 93% with only 2.29K parameters.

Related papers

• Gesture Recognition for FMCW Radar on the Edge [0.0]
  We show that gestures can be characterized efficiently by a set of five features. A recurrent neural network (RNN) based architecture exploits these features to jointly detect and classify five different gestures. The proposed system recognizes gestures with an F1 score of 98.4% on our hold-out test dataset.
  arXiv  Detail & Related papers  (2023-10-13T06:03:07Z)
• UnLoc: A Universal Localization Method for Autonomous Vehicles using LiDAR, Radar and/or Camera Input [51.150605800173366]
  UnLoc is a novel unified neural modeling approach for localization with multi-sensor input in all weather conditions. Our method is extensively evaluated on Oxford Radar RobotCar, ApolloSouthBay and Perth-WA datasets.
  arXiv  Detail & Related papers  (2023-07-03T04:10:55Z)
• Agile gesture recognition for capacitive sensing devices: adapting on-the-job [55.40855017016652]
  We demonstrate a hand gesture recognition system that uses signals from capacitive sensors embedded into the etee hand controller. The controller generates real-time signals from each of the wearer five fingers. We use a machine learning technique to analyse the time series signals and identify three features that can represent 5 fingers within 500 ms.
  arXiv  Detail & Related papers  (2023-05-12T17:24:02Z)
• It Takes Two: Masked Appearance-Motion Modeling for Self-supervised Video Transformer Pre-training [76.69480467101143]
  Self-supervised video transformer pre-training has recently benefited from the mask-and-predict pipeline. We explicitly investigate motion cues in videos as extra prediction target and propose our Masked Appearance-Motion Modeling framework. Our method learns generalized video representations and achieves 82.3% on Kinects-400, 71.3% on Something-Something V2, 91.5% on UCF101, and 62.5% on HMDB51.
  arXiv  Detail & Related papers  (2022-10-11T08:05:18Z)
• A Wireless-Vision Dataset for Privacy Preserving Human Activity Recognition [53.41825941088989]
  A new WiFi-based and video-based neural network (WiNN) is proposed to improve the robustness of activity recognition. Our results show that WiVi data set satisfies the primary demand and all three branches in the proposed pipeline keep more than $80%$ of activity recognition accuracy.
  arXiv  Detail & Related papers  (2022-05-24T10:49:11Z)
• iSDF: Real-Time Neural Signed Distance Fields for Robot Perception [64.80458128766254]
  iSDF is a continuous learning system for real-time signed distance field reconstruction. It produces more accurate reconstructions and better approximations of collision costs and gradients.
  arXiv  Detail & Related papers  (2022-04-05T15:48:39Z)
• A Prospective Approach for Human-to-Human Interaction Recognition from Wi-Fi Channel Data using Attention Bidirectional Gated Recurrent Neural Network with GUI Application Implementation [0.0]
  This study uses a Multiple Input Multiple Output radio link between a WiFi router and an Intel 5300 NIC, with the time-series Wi-Fi channel state information for mutual human-to-human concurrent interaction recognition. The proposed Self-Attention guided Bidirectional Gated Recurrent Neural Network can classify 13 mutual interactions with a maximum benchmark accuracy of 94% for a single subject-pair.
  arXiv  Detail & Related papers  (2022-02-16T15:40:52Z)
• Towards Domain-Independent and Real-Time Gesture Recognition Using mmWave Signal [11.76969975145963]
  DI-Gesture is a domain-independent and real-time mmWave gesture recognition system. In real-time scenario, the accuracy of DI-Gesutre reaches over 97% with average inference time of 2.87ms.
  arXiv  Detail & Related papers  (2021-11-11T13:28:28Z)
• Tesla-Rapture: A Lightweight Gesture Recognition System from mmWave Radar Point Clouds [28.304829106423988]
  Tesla-Rapture is a gesture recognition interface for point clouds generated by mmWave Radars. We develop Tesla, a Message Passing Neural Network (MPNN) graph convolution approach for mmWave radar point clouds. We publish the source code, the trained models, and the implementation of the model for embedded devices.
  arXiv  Detail & Related papers  (2021-09-14T05:25:17Z)
• RT3D: Achieving Real-Time Execution of 3D Convolutional Neural Networks on Mobile Devices [57.877112704841366]
  This paper proposes RT3D, a model compression and mobile acceleration framework for 3D CNNs. For the first time, real-time execution of 3D CNNs is achieved on off-the-shelf mobiles.
  arXiv  Detail & Related papers  (2020-07-20T02:05:32Z)
• TinyRadarNN: Combining Spatial and Temporal Convolutional Neural Networks for Embedded Gesture Recognition with Short Range Radars [13.266626571886354]
  This work proposes a low-power high-accuracy embedded hand-gesture recognition algorithm targeting battery-operated wearable devices. A 2D Convolutional Neural Network (CNN) using range frequency Doppler features is combined with a Temporal Convolutional Neural Network (TCN) for time sequence prediction.
  arXiv  Detail & Related papers  (2020-06-25T15:23:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.