Fusing Hand and Body Skeletons for Human Action Recognition in Assembly
- URL: http://arxiv.org/abs/2307.09238v1
- Date: Tue, 18 Jul 2023 13:18:52 GMT
- Title: Fusing Hand and Body Skeletons for Human Action Recognition in Assembly
- Authors: Dustin Aganian, Mona K\"ohler, Benedict Stephan, Markus Eisenbach,
Horst-Michael Gross
- Abstract summary: We propose a method in which less detailed body skeletons are combined with highly detailed hand skeletons.
This paper demonstrates the effectiveness of our proposed approach in enhancing action recognition in assembly scenarios.
- Score: 13.24875937437949
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: As collaborative robots (cobots) continue to gain popularity in industrial
manufacturing, effective human-robot collaboration becomes crucial. Cobots
should be able to recognize human actions to assist with assembly tasks and act
autonomously. To achieve this, skeleton-based approaches are often used due to
their ability to generalize across various people and environments. Although
body skeleton approaches are widely used for action recognition, they may not
be accurate enough for assembly actions where the worker's fingers and hands
play a significant role. To address this limitation, we propose a method in
which less detailed body skeletons are combined with highly detailed hand
skeletons. We investigate CNNs and transformers, the latter of which are
particularly adept at extracting and combining important information from both
skeleton types using attention. This paper demonstrates the effectiveness of
our proposed approach in enhancing action recognition in assembly scenarios.
Related papers
- SkeleTR: Towrads Skeleton-based Action Recognition in the Wild [86.03082891242698]
SkeleTR is a new framework for skeleton-based action recognition.
It first models the intra-person skeleton dynamics for each skeleton sequence with graph convolutions.
It then uses stacked Transformer encoders to capture person interactions that are important for action recognition in general scenarios.
arXiv Detail & Related papers (2023-09-20T16:22:33Z) - Dynamic Hand Gesture-Featured Human Motor Adaptation in Tool Delivery
using Voice Recognition [5.13619372598999]
This paper introduces an innovative human-robot collaborative framework.
It seamlessly integrates hand gesture and dynamic movement recognition, voice recognition, and a switchable control adaptation strategy.
Experiment results have demonstrated superior performance in hand gesture recognition.
arXiv Detail & Related papers (2023-09-20T14:51:09Z) - Giving Robots a Hand: Learning Generalizable Manipulation with
Eye-in-Hand Human Video Demonstrations [66.47064743686953]
Eye-in-hand cameras have shown promise in enabling greater sample efficiency and generalization in vision-based robotic manipulation.
Videos of humans performing tasks, on the other hand, are much cheaper to collect since they eliminate the need for expertise in robotic teleoperation.
In this work, we augment narrow robotic imitation datasets with broad unlabeled human video demonstrations to greatly enhance the generalization of eye-in-hand visuomotor policies.
arXiv Detail & Related papers (2023-07-12T07:04:53Z) - How Object Information Improves Skeleton-based Human Action Recognition
in Assembly Tasks [12.349172146831506]
We present a novel approach of integrating object information into skeleton-based action recognition.
We enhance two state-of-the-art methods by treating object centers as further skeleton joints.
Our research sheds light on the benefits of combining skeleton joints with object information for human action recognition in assembly tasks.
arXiv Detail & Related papers (2023-06-09T12:18:14Z) - Joint Action is a Framework for Understanding Partnerships Between
Humans and Upper Limb Prostheses [0.6649973446180738]
We compare different prosthesis controllers (proportional electromyography with sequential switching, pattern recognition, and adaptive switching) in terms of how they present the hallmarks of joint action.
The results of the comparison lead to a new perspective for understanding how existing myoelectric systems relate to each other.
arXiv Detail & Related papers (2022-12-28T23:27:32Z) - Generalizable Human-Robot Collaborative Assembly Using Imitation
Learning and Force Control [17.270360447188196]
We present a system for human-robot collaborative assembly using learning from demonstration and pose estimation.
The proposed system is demonstrated using a physical 6 DoF manipulator in a collaborative human-robot assembly scenario.
arXiv Detail & Related papers (2022-12-02T20:35:55Z) - Skeleton-Based Mutually Assisted Interacted Object Localization and
Human Action Recognition [111.87412719773889]
We propose a joint learning framework for "interacted object localization" and "human action recognition" based on skeleton data.
Our method achieves the best or competitive performance with the state-of-the-art methods for human action recognition.
arXiv Detail & Related papers (2021-10-28T10:09:34Z) - Show Me What You Can Do: Capability Calibration on Reachable Workspace
for Human-Robot Collaboration [83.4081612443128]
We show that a short calibration using REMP can effectively bridge the gap between what a non-expert user thinks a robot can reach and the ground-truth.
We show that this calibration procedure not only results in better user perception, but also promotes more efficient human-robot collaborations.
arXiv Detail & Related papers (2021-03-06T09:14:30Z) - Skeleton-Aware Networks for Deep Motion Retargeting [83.65593033474384]
We introduce a novel deep learning framework for data-driven motion between skeletons.
Our approach learns how to retarget without requiring any explicit pairing between the motions in the training set.
arXiv Detail & Related papers (2020-05-12T12:51:40Z) - Human Grasp Classification for Reactive Human-to-Robot Handovers [50.91803283297065]
We propose an approach for human-to-robot handovers in which the robot meets the human halfway.
We collect a human grasp dataset which covers typical ways of holding objects with various hand shapes and poses.
We present a planning and execution approach that takes the object from the human hand according to the detected grasp and hand position.
arXiv Detail & Related papers (2020-03-12T19:58:03Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.