Wearable Sensor-Based Few-Shot Continual Learning on Hand Gestures for Motor-Impaired Individuals via Latent Embedding Exploitation

• URL: http://arxiv.org/abs/2405.08969v2
• Date: Tue, 11 Jun 2024 20:33:20 GMT
• Title: Wearable Sensor-Based Few-Shot Continual Learning on Hand Gestures for Motor-Impaired Individuals via Latent Embedding Exploitation
• Authors: Riyad Bin Rafiq, Weishi Shi, Mark V. Albert,
• Abstract summary: We introduce the Latent Embedding Exploitation (LEE) mechanism in our replay-based Few-Shot Continual Learning framework. Our method produces a diversified latent feature space by leveraging a preserved latent embedding known as gesture prior knowledge. Our method helps motor-impaired persons leverage wearable devices, and their unique styles of movement can be learned and applied.
• Score: 6.782362178252351
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Hand gestures can provide a natural means of human-computer interaction and enable people who cannot speak to communicate efficiently. Existing hand gesture recognition methods heavily depend on pre-defined gestures, however, motor-impaired individuals require new gestures tailored to each individual's gesture motion and style. Gesture samples collected from different persons have distribution shifts due to their health conditions, the severity of the disability, motion patterns of the arms, etc. In this paper, we introduce the Latent Embedding Exploitation (LEE) mechanism in our replay-based Few-Shot Continual Learning (FSCL) framework that significantly improves the performance of fine-tuning a model for out-of-distribution data. Our method produces a diversified latent feature space by leveraging a preserved latent embedding known as gesture prior knowledge, along with intra-gesture divergence derived from two additional embeddings. Thus, the model can capture latent statistical structure in highly variable gestures with limited samples. We conduct an experimental evaluation using the SmartWatch Gesture and the Motion Gesture datasets. The proposed method results in an average test accuracy of 57.0%, 64.6%, and 69.3% by using one, three, and five samples for six different gestures. Our method helps motor-impaired persons leverage wearable devices, and their unique styles of movement can be learned and applied in human-computer interaction and social communication. Code is available at: https://github.com/riyadRafiq/wearable-latent-embedding-exploitation

Related papers

• ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesis [50.69464138626748]
  We present ConvoFusion, a diffusion-based approach for multi-modal gesture synthesis. Our method proposes two guidance objectives that allow the users to modulate the impact of different conditioning modalities. Our method is versatile in that it can be trained either for generating monologue gestures or even the conversational gestures.
  arXiv  Detail & Related papers  (2024-03-26T17:59:52Z)
• InterControl: Zero-shot Human Interaction Generation by Controlling Every Joint [67.6297384588837]
  We introduce a novel controllable motion generation method, InterControl, to encourage the synthesized motions maintaining the desired distance between joint pairs. We demonstrate that the distance between joint pairs for human-wise interactions can be generated using an off-the-shelf Large Language Model.
  arXiv  Detail & Related papers  (2023-11-27T14:32:33Z)
• Reconfigurable Data Glove for Reconstructing Physical and Virtual Grasps [100.72245315180433]
  We present a reconfigurable data glove design to capture different modes of human hand-object interactions. The glove operates in three modes for various downstream tasks with distinct features. We evaluate the system's three modes by (i) recording hand gestures and associated forces, (ii) improving manipulation fluency in VR, and (iii) producing realistic simulation effects of various tool uses.
  arXiv  Detail & Related papers  (2023-01-14T05:35:50Z)
• Fast Learning of Dynamic Hand Gesture Recognition with Few-Shot Learning Models [0.0]
  We develop Few-Shot Learning models trained to recognize five or ten different dynamic hand gestures. Models are arbitrarily interchangeable by providing the model with one, two, or five examples per hand gesture. Result show accuracy of up to 88.8% for recognition of five and up to 81.2% for ten dynamic hand gestures.
  arXiv  Detail & Related papers  (2022-12-16T09:31:15Z)
• Enabling hand gesture customization on wrist-worn devices [28.583516259577486]
  We present a framework for gesture customization requiring minimal examples from users, all without degrading the performance of existing gesture sets. Our approach paves the way for a future where users are no longer bound to pre-existing gestures, freeing them to creatively introduce new gestures tailored to their preferences and abilities.
  arXiv  Detail & Related papers  (2022-03-29T05:12:32Z)
• Dynamic Modeling of Hand-Object Interactions via Tactile Sensing [133.52375730875696]
  In this work, we employ a high-resolution tactile glove to perform four different interactive activities on a diversified set of objects. We build our model on a cross-modal learning framework and generate the labels using a visual processing pipeline to supervise the tactile model. This work takes a step on dynamics modeling in hand-object interactions from dense tactile sensing.
  arXiv  Detail & Related papers  (2021-09-09T16:04:14Z)
• What Matters in Learning from Offline Human Demonstrations for Robot Manipulation [64.43440450794495]
  We conduct an extensive study of six offline learning algorithms for robot manipulation. Our study analyzes the most critical challenges when learning from offline human data. We highlight opportunities for learning from human datasets.
  arXiv  Detail & Related papers  (2021-08-06T20:48:30Z)
• Improving Human Motion Prediction Through Continual Learning [2.720960618356385]
  Human motion prediction is an essential component for enabling closer human-robot collaboration. It is compounded by the variability of human motion, both at a skeletal level due to the varying size of humans and at a motion level due to individual movement idiosyncrasies. We propose a modular sequence learning approach that allows end-to-end training while also having the flexibility of being fine-tuned.
  arXiv  Detail & Related papers  (2021-07-01T15:34:41Z)
• Multi-modal Fusion for Single-Stage Continuous Gesture Recognition [45.19890687786009]
  We introduce a single-stage continuous gesture recognition framework, called Temporal Multi-Modal Fusion (TMMF) TMMF can detect and classify multiple gestures in a video via a single model. This approach learns the natural transitions between gestures and non-gestures without the need for a pre-processing segmentation step.
  arXiv  Detail & Related papers  (2020-11-10T07:09:35Z)
• Relational Graph Learning on Visual and Kinematics Embeddings for Accurate Gesture Recognition in Robotic Surgery [84.73764603474413]
  We propose a novel online approach of multi-modal graph network (i.e., MRG-Net) to dynamically integrate visual and kinematics information. The effectiveness of our method is demonstrated with state-of-the-art results on the public JIGSAWS dataset.
  arXiv  Detail & Related papers  (2020-11-03T11:00:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.