Human Part-wise 3D Motion Context Learning for Sign Language Recognition

• URL: http://arxiv.org/abs/2308.09305v1
• Date: Fri, 18 Aug 2023 05:01:52 GMT
• Title: Human Part-wise 3D Motion Context Learning for Sign Language Recognition
• Abstract summary: We propose P3D, the human part-wise motion context learning framework for sign language recognition. Our main contributions lie in two dimensions: learning the part-wise motion context and employing the pose ensemble to utilize 2D and 3D pose jointly.
• Score: 54.772707210595065
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: In this paper, we propose P3D, the human part-wise motion context learning framework for sign language recognition. Our main contributions lie in two dimensions: learning the part-wise motion context and employing the pose ensemble to utilize 2D and 3D pose jointly. First, our empirical observation implies that part-wise context encoding benefits the performance of sign language recognition. While previous methods of sign language recognition learned motion context from the sequence of the entire pose, we argue that such methods cannot exploit part-specific motion context. In order to utilize part-wise motion context, we propose the alternating combination of a part-wise encoding Transformer (PET) and a whole-body encoding Transformer (WET). PET encodes the motion contexts from a part sequence, while WET merges them into a unified context. By learning part-wise motion context, our P3D achieves superior performance on WLASL compared to previous state-of-the-art methods. Second, our framework is the first to ensemble 2D and 3D poses for sign language recognition. Since the 3D pose holds rich motion context and depth information to distinguish the words, our P3D outperformed the previous state-of-the-art methods employing a pose ensemble.

Related papers

• Fine-grained Human Motion Understanding with Language Models [11.539750714535773]
  Methodname represents motion as a sequence of skeletal poses with explicit timestamps for each pose.<n>We construct a training mixture spanning pose captioning, pose question answering, motion captioning, and motion question answering.<n>Our approach supports both 2D and 3D skeletal motion representations through a unified pose encoder.
  arXiv  Detail & Related papers  (2026-06-18T19:31:56Z)
• 2D or not 2D: How Does the Dimensionality of Gesture Representation Affect 3D Co-Speech Gesture Generation? [5.408549711581793]
  We study the effect of using either 2D or 3D joint coordinates as training data on the performance of speech-to-gesture deep generative models. We employ a lifting model for converting generated 2D pose sequences into 3D and assess how gestures created directly in 3D stack up against those initially generated in 2D and then converted to 3D.
  arXiv  Detail & Related papers  (2024-09-16T15:06:12Z)
• GHOST: Grounded Human Motion Generation with Open Vocabulary Scene-and-Text Contexts [48.28000728061778]
  We propose a method that integrates an open vocabulary scene encoder into the architecture, establishing a robust connection between text and scene. Our methodology achieves up to a 30% reduction in the goal object distance metric compared to the prior state-of-the-art baseline model.
  arXiv  Detail & Related papers  (2024-04-08T18:24:12Z)
• Sign Language Production with Latent Motion Transformer [2.184775414778289]
  We develop a new method to make high-quality sign videos without using human poses as a middle step. Our model works in two main parts: first, it learns from a generator and the video's hidden features, and next, it uses another model to understand the order of these hidden features. Compared with previous state-of-the-art approaches, our model performs consistently better on two word-level sign language datasets.
  arXiv  Detail & Related papers  (2023-12-20T10:53:06Z)
• Sculpting Holistic 3D Representation in Contrastive Language-Image-3D Pre-training [51.632418297156605]
  We introduce MixCon3D, a method aiming to sculpt holistic 3D representation in contrastive language-image-3D pre-training. We develop the 3D object-level representation from complementary perspectives, e.g., multi-view rendered images with the point cloud. Then, MixCon3D performs language-3D contrastive learning, comprehensively depicting real-world 3D objects and bolstering text alignment.
  arXiv  Detail & Related papers  (2023-11-03T06:05:36Z)
• CLIP-Hand3D: Exploiting 3D Hand Pose Estimation via Context-Aware Prompting [38.678165053219644]
  We make one of the first attempts to propose a novel 3D hand pose estimator from monocular images, dubbed as CLIP-Hand3D. We maximize semantic consistency for a pair of pose-text features following a CLIP-based contrastive learning paradigm. Experiments on several public hand benchmarks show that the proposed model attains a significantly faster inference speed.
  arXiv  Detail & Related papers  (2023-09-28T03:40:37Z)
• Reconstructing Signing Avatars From Video Using Linguistic Priors [54.5282429129769]
  Sign language (SL) is the primary method of communication for the 70 million Deaf people around the world. replacing video dictionaries of isolated signs with 3D avatars can aid learning and enable AR/VR applications. SGNify captures fine-grained hand pose, facial expression, and body movement fully automatically from in-the-wild monocular SL videos.
  arXiv  Detail & Related papers  (2023-04-20T17:29:50Z)
• Learning Speech-driven 3D Conversational Gestures from Video [106.15628979352738]
  We propose the first approach to automatically and jointly synthesize both the synchronous 3D conversational body and hand gestures. Our algorithm uses a CNN architecture that leverages the inherent correlation between facial expression and hand gestures. We also contribute a new way to create a large corpus of more than 33 hours of annotated body, hand, and face data from in-the-wild videos of talking people.
  arXiv  Detail & Related papers  (2021-02-13T01:05:39Z)
• Depth-Aware Action Recognition: Pose-Motion Encoding through Temporal Heatmaps [2.2079886535603084]
  We propose a depth-aware descriptor that encodes pose and motion information in a unified representation for action classification in-the-wild. The key component of our method is the Depth-Aware Pose Motion representation (DA-PoTion), a new video descriptor that encodes the 3D movement of semantic keypoints of the human body.
  arXiv  Detail & Related papers  (2020-11-26T17:26:42Z)
• Body2Hands: Learning to Infer 3D Hands from Conversational Gesture Body Dynamics [87.17505994436308]
  We build upon the insight that body motion and hand gestures are strongly correlated in non-verbal communication settings. We formulate the learning of this prior as a prediction task of 3D hand shape over time given body motion input alone. Our hand prediction model produces convincing 3D hand gestures given only the 3D motion of the speaker's arms as input.
  arXiv  Detail & Related papers  (2020-07-23T22:58:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.