Local Spherical Harmonics Improve Skeleton-Based Hand Action Recognition

• URL: http://arxiv.org/abs/2308.10557v2
• Date: Tue, 14 Nov 2023 12:20:02 GMT
• Title: Local Spherical Harmonics Improve Skeleton-Based Hand Action Recognition
• Authors: Katharina Prasse, Steffen Jung, Yuxuan Zhou, Margret Keuper
• Abstract summary: We propose a method specifically designed for hand action recognition which uses relative angular embeddings and local Spherical Harmonics to create novel hand representations. The use of Spherical Harmonics creates rotation-invariant representations which make hand action recognition even more robust against inter-subject differences and viewpoint changes.
• Score: 17.62840662799232
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Hand action recognition is essential. Communication, human-robot interactions, and gesture control are dependent on it. Skeleton-based action recognition traditionally includes hands, which belong to the classes which remain challenging to correctly recognize to date. We propose a method specifically designed for hand action recognition which uses relative angular embeddings and local Spherical Harmonics to create novel hand representations. The use of Spherical Harmonics creates rotation-invariant representations which make hand action recognition even more robust against inter-subject differences and viewpoint changes. We conduct extensive experiments on the hand joints in the First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations, and on the NTU RGB+D 120 dataset, demonstrating the benefit of using Local Spherical Harmonics Representations. Our code is available at https://github.com/KathPra/LSHR_LSHT.

Related papers

• On the Utility of 3D Hand Poses for Action Recognition [36.64538554919222]
  HandFormer is a novel multimodal transformer to efficiently model hand-object interactions. We factorize hand modeling and represent each joint by its short-term trajectories. We achieve new state-of-the-art performance on Assembly101 and H2O with significant improvements in egocentric action recognition.
  arXiv  Detail & Related papers  (2024-03-14T18:52:34Z)
• GRIP: Generating Interaction Poses Using Spatial Cues and Latent Consistency [57.9920824261925]
  Hands are dexterous and highly versatile manipulators that are central to how humans interact with objects and their environment. modeling realistic hand-object interactions is critical for applications in computer graphics, computer vision, and mixed reality. GRIP is a learning-based method that takes as input the 3D motion of the body and the object, and synthesizes realistic motion for both hands before, during, and after object interaction.
  arXiv  Detail & Related papers  (2023-08-22T17:59:51Z)
• HandNeRF: Neural Radiance Fields for Animatable Interacting Hands [122.32855646927013]
  We propose a novel framework to reconstruct accurate appearance and geometry with neural radiance fields (NeRF) for interacting hands. We conduct extensive experiments to verify the merits of our proposed HandNeRF and report a series of state-of-the-art results.
  arXiv  Detail & Related papers  (2023-03-24T06:19:19Z)
• Im2Hands: Learning Attentive Implicit Representation of Interacting Two-Hand Shapes [58.551154822792284]
  Implicit Two Hands (Im2Hands) is the first neural implicit representation of two interacting hands. Im2Hands can produce fine-grained geometry of two hands with high hand-to-hand and hand-to-image coherency. We experimentally demonstrate the effectiveness of Im2Hands on two-hand reconstruction in comparison to related methods.
  arXiv  Detail & Related papers  (2023-02-28T06:38:25Z)
• Generative Action Description Prompts for Skeleton-based Action Recognition [15.38417530693649]
  We propose a Generative Action-description Prompts (GAP) approach for skeleton-based action recognition. We employ a pre-trained large-scale language model as the knowledge engine to automatically generate text descriptions for body parts movements of actions. Our proposed GAP method achieves noticeable improvements over various baseline models without extra cost at inference.
  arXiv  Detail & Related papers  (2022-08-10T12:55:56Z)
• Monocular 3D Reconstruction of Interacting Hands via Collision-Aware Factorized Refinements [96.40125818594952]
  We make the first attempt to reconstruct 3D interacting hands from monocular single RGB images. Our method can generate 3D hand meshes with both precise 3D poses and minimal collisions.
  arXiv  Detail & Related papers  (2021-11-01T08:24:10Z)
• On-device Real-time Hand Gesture Recognition [1.4658400971135652]
  We present an on-device real-time hand gesture recognition (HGR) system, which detects a set of predefined static gestures from a single RGB camera. We use MediaPipe Hands as the basis of the hand skeleton tracker, improve the keypoint accuracy, and add the estimation of 3D keypoints in a world metric space.
  arXiv  Detail & Related papers  (2021-10-29T18:33:25Z)
• Egocentric View Hand Action Recognition by Leveraging Hand Surface and Hand Grasp Type [15.878905144552204]
  The framework synthesizes the mean curvature of the hand mesh model to encode the hand surface geometry in 3D space. Using hand grasp type and mean curvature of hand increases the performance of the hand action recognition.
  arXiv  Detail & Related papers  (2021-09-08T17:12:02Z)
• RGB2Hands: Real-Time Tracking of 3D Hand Interactions from Monocular RGB Video [76.86512780916827]
  We present the first real-time method for motion capture of skeletal pose and 3D surface geometry of hands from a single RGB camera. In order to address the inherent depth ambiguities in RGB data, we propose a novel multi-task CNN. We experimentally verify the individual components of our RGB two-hand tracking and 3D reconstruction pipeline.
  arXiv  Detail & Related papers  (2021-06-22T12:53:56Z)
• SHREC 2021: Track on Skeleton-based Hand Gesture Recognition in the Wild [62.450907796261646]
  Recognition of hand gestures can be performed directly from the stream of hand skeletons estimated by software. Despite the recent advancements in gesture and action recognition from skeletons, it is unclear how well the current state-of-the-art techniques can perform in a real-world scenario. This paper presents the results of the SHREC 2021: Track on Skeleton-based Hand Gesture Recognition in the Wild contest.
  arXiv  Detail & Related papers  (2021-06-21T10:57:49Z)
• FineHand: Learning Hand Shapes for American Sign Language Recognition [16.862375555609667]
  We present an approach for effective learning of hand shape embeddings, which are discriminative for ASL gestures. For hand shape recognition our method uses a mix of manually labelled hand shapes and high confidence predictions to train deep convolutional neural network (CNN) We will demonstrate that higher quality hand shape models can significantly improve the accuracy of final video gesture classification.
  arXiv  Detail & Related papers  (2020-03-04T23:32:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.