Recognizing Hand Use and Hand Role at Home After Stroke from Egocentric Video

• URL: http://arxiv.org/abs/2207.08920v2
• Date: Thu, 21 Jul 2022 16:02:22 GMT
• Title: Recognizing Hand Use and Hand Role at Home After Stroke from Egocentric Video
• Authors: Meng-Fen Tsai, Rosalie H. Wang, and Jo\'se Zariffa
• Abstract summary: Egocentric video can capture hand-object interactions in context, as well as show how more-affected hands are used. To use artificial intelligence-based computer vision to classify hand use and hand role from egocentric videos recorded at home after stroke.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Introduction: Hand function is a central determinant of independence after stroke. Measuring hand use in the home environment is necessary to evaluate the impact of new interventions, and calls for novel wearable technologies. Egocentric video can capture hand-object interactions in context, as well as show how more-affected hands are used during bilateral tasks (for stabilization or manipulation). Automated methods are required to extract this information. Objective: To use artificial intelligence-based computer vision to classify hand use and hand role from egocentric videos recorded at home after stroke. Methods: Twenty-one stroke survivors participated in the study. A random forest classifier, a SlowFast neural network, and the Hand Object Detector neural network were applied to identify hand use and hand role at home. Leave-One-Subject-Out-Cross-Validation (LOSOCV) was used to evaluate the performance of the three models. Between-group differences of the models were calculated based on the Mathews correlation coefficient (MCC). Results: For hand use detection, the Hand Object Detector had significantly higher performance than the other models. The macro average MCCs using this model in the LOSOCV were 0.50 +- 0.23 for the more-affected hands and 0.58 +- 0.18 for the less-affected hands. Hand role classification had macro average MCCs in the LOSOCV that were close to zero for all models. Conclusion: Using egocentric video to capture the hand use of stroke survivors at home is feasible. Pose estimation to track finger movements may be beneficial to classifying hand roles in the future.

Related papers

• Testing Human-Hand Segmentation on In-Distribution and Out-of-Distribution Data in Human-Robot Interactions Using a Deep Ensemble Model [40.815678328617686]
  We present a novel approach by evaluating the performance of pre-trained deep learning models under both ID data and more challenging OOD scenarios. We incorporated unique and rare conditions such as finger-crossing gestures and motion blur from fast-moving hands. Results revealed that models trained on industrial datasets outperformed those trained on non-industrial datasets.
  arXiv  Detail & Related papers  (2025-01-13T21:52:46Z)
• Detecting Activities of Daily Living in Egocentric Video to Contextualize Hand Use at Home in Outpatient Neurorehabilitation Settings [2.9158689853305693]
  We show that an object-centric approach, focusing on what objects patients interact with rather than how they move, can effectively recognize Activities of Daily Living (ADL) in real-world rehabilitation settings. We evaluated our models on a complex dataset collected in the wild comprising 2261 minutes of egocentric video from 16 participants with impaired hand function.
  arXiv  Detail & Related papers  (2024-12-14T14:38:27Z)
• A Personalized Video-Based Hand Taxonomy: Application for Individuals with Spinal Cord Injury [14.062874246796687]
  Spinal cord injuries (SCI) can impair hand function, reducing independence. This study aims to automatically identify the dominant distinct hand grasps in egocentric video using semantic clustering. A deep learning model integrating posture and appearance data was employed to create a personalized hand taxonomy.
  arXiv  Detail & Related papers  (2024-03-26T20:30:55Z)
• HMP: Hand Motion Priors for Pose and Shape Estimation from Video [52.39020275278984]
  We develop a generative motion prior specific for hands, trained on the AMASS dataset which features diverse and high-quality hand motions. Our integration of a robust motion prior significantly enhances performance, especially in occluded scenarios. We demonstrate our method's efficacy via qualitative and quantitative evaluations on the HO3D and DexYCB datasets.
  arXiv  Detail & Related papers  (2023-12-27T22:35:33Z)
• Semantics2Hands: Transferring Hand Motion Semantics between Avatars [34.39785320233128]
  Even minor errors in hand motions can significantly impact the user experience. This paper introduces a novel anatomy-based semantic matrix (ASM) that encodes the semantics of hand motions. We train the ASM using a semi-supervised learning strategy on the Mixamo and InterHand2.6M datasets.
  arXiv  Detail & Related papers  (2023-08-11T03:07:31Z)
• Interacting Hand-Object Pose Estimation via Dense Mutual Attention [97.26400229871888]
  3D hand-object pose estimation is the key to the success of many computer vision applications. We propose a novel dense mutual attention mechanism that is able to model fine-grained dependencies between the hand and the object. Our method is able to produce physically plausible poses with high quality and real-time inference speed.
  arXiv  Detail & Related papers  (2022-11-16T10:01:33Z)
• Measuring hand use in the home after cervical spinal cord injury using egocentric video [2.1064122195521926]
  Egocentric video has emerged as a potential solution for monitoring hand function in individuals living with tetraplegia in the community. We develop and validate a wearable vision-based system for measuring hand use in the home among individuals living with tetraplegia.
  arXiv  Detail & Related papers  (2022-03-31T12:43:23Z)
• Monocular 3D Reconstruction of Interacting Hands via Collision-Aware Factorized Refinements [96.40125818594952]
  We make the first attempt to reconstruct 3D interacting hands from monocular single RGB images. Our method can generate 3D hand meshes with both precise 3D poses and minimal collisions.
  arXiv  Detail & Related papers  (2021-11-01T08:24:10Z)
• A Skeleton-Driven Neural Occupancy Representation for Articulated Hands [49.956892429789775]
  Hand ArticuLated Occupancy (HALO) is a novel representation of articulated hands that bridges the advantages of 3D keypoints and neural implicit surfaces. We demonstrate the applicability of HALO to the task of conditional generation of hands that grasp 3D objects.
  arXiv  Detail & Related papers  (2021-09-23T14:35:19Z)
• Domain Adaptive Robotic Gesture Recognition with Unsupervised Kinematic-Visual Data Alignment [60.31418655784291]
  We propose a novel unsupervised domain adaptation framework which can simultaneously transfer multi-modality knowledge, i.e., both kinematic and visual data, from simulator to real robot. It remedies the domain gap with enhanced transferable features by using temporal cues in videos, and inherent correlations in multi-modal towards recognizing gesture. Results show that our approach recovers the performance with great improvement gains, up to 12.91% in ACC and 20.16% in F1score without using any annotations in real robot.
  arXiv  Detail & Related papers  (2021-03-06T09:10:03Z)
• Where is my hand? Deep hand segmentation for visual self-recognition in humanoid robots [129.46920552019247]
  We propose the use of a Convolution Neural Network (CNN) to segment the robot hand from an image in an egocentric view. We fine-tuned the Mask-RCNN network for the specific task of segmenting the hand of the humanoid robot Vizzy.
  arXiv  Detail & Related papers  (2021-02-09T10:34:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.