Sign Language Sense Disambiguation

• URL: http://arxiv.org/abs/2409.08780v1
• Date: Fri, 13 Sep 2024 12:36:52 GMT
• Title: Sign Language Sense Disambiguation
• Authors: Jana Grimm, Miriam Winkler, Oliver Kraus, Tanalp Agustoslu,
• Abstract summary: This project explores methods to enhance sign language translation of German sign language, specifically focusing on homonyms. We approach the improvement by training transformer-based models on various bodypart representations to shift the focus on said bodypart. The results show that focusing on the mouth increases the performance in small dataset settings while shifting the focus on the hands retrieves better results in larger dataset settings.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This project explores methods to enhance sign language translation of German sign language, specifically focusing on disambiguation of homonyms. Sign language is ambiguous and understudied which is the basis for our experiments. We approach the improvement by training transformer-based models on various bodypart representations to shift the focus on said bodypart. To determine the impact of, e.g., the hand or mouth representations, we experiment with different combinations. The results show that focusing on the mouth increases the performance in small dataset settings while shifting the focus on the hands retrieves better results in larger dataset settings. Our results contribute to better accessibility for non-hearing persons by improving the systems powering digital assistants, enabling a more accurate interaction. The code for this project can be found on GitHub.

Related papers

• EvSign: Sign Language Recognition and Translation with Streaming Events [59.51655336911345]
  Event camera could naturally perceive dynamic hand movements, providing rich manual clues for sign language tasks. We propose efficient transformer-based framework for event-based SLR and SLT tasks. Our method performs favorably against existing state-of-the-art approaches with only 0.34% computational cost.
  arXiv  Detail & Related papers  (2024-07-17T14:16:35Z)
• A Transformer-Based Multi-Stream Approach for Isolated Iranian Sign Language Recognition [0.0]
  This research aims to recognize Iranian Sign Language words with the help of the latest deep learning tools such as transformers. The dataset used includes 101 Iranian Sign Language words frequently used in academic environments such as universities.
  arXiv  Detail & Related papers  (2024-06-27T06:54:25Z)
• Self-Supervised Representation Learning with Spatial-Temporal Consistency for Sign Language Recognition [96.62264528407863]
  We propose a self-supervised contrastive learning framework to excavate rich context via spatial-temporal consistency. Inspired by the complementary property of motion and joint modalities, we first introduce first-order motion information into sign language modeling. Our method is evaluated with extensive experiments on four public benchmarks, and achieves new state-of-the-art performance with a notable margin.
  arXiv  Detail & Related papers  (2024-06-15T04:50:19Z)
• SignMusketeers: An Efficient Multi-Stream Approach for Sign Language Translation at Scale [22.49602248323602]
  A persistent challenge in sign language video processing is how we learn representations of sign language. Our proposed method focuses on just the most relevant parts in a signing video: the face, hands and body posture of the signer. Our approach is based on learning from individual frames (rather than video sequences) and is therefore much more efficient than prior work on sign language pre-training.
  arXiv  Detail & Related papers  (2024-06-11T03:00:41Z)
• Roles of Scaling and Instruction Tuning in Language Perception: Model vs. Human Attention [58.817405319722596]
  This work compares the self-attention of several large language models (LLMs) in different sizes to assess the effect of scaling and instruction tuning on language perception. Results show that scaling enhances the human resemblance and improves the effective attention by reducing the trivial pattern reliance, while instruction tuning does not. We also find that current LLMs are consistently closer to non-native than native speakers in attention, suggesting a sub-optimal language perception of all models.
  arXiv  Detail & Related papers  (2023-10-29T17:16:40Z)
• Building Korean Sign Language Augmentation (KoSLA) Corpus with Data Augmentation Technique [0.0]
  We present an efficient framework of corpus for sign language translation. By considering the linguistic features of sign language, our proposed framework is a first and unique attempt to build a multimodal sign language augmentation corpus.
  arXiv  Detail & Related papers  (2022-07-12T02:12:36Z)
• Keypoint based Sign Language Translation without Glosses [7.240731862549344]
  We propose a new keypoint normalization method for performing translation based on the skeleton point of the signer. It contributed to performance improvement by a customized normalization method depending on the body parts. Our method can be applied to various datasets in a way that can be applied to datasets without glosses.
  arXiv  Detail & Related papers  (2022-04-22T05:37:56Z)
• XDBERT: Distilling Visual Information to BERT from Cross-Modal Systems to Improve Language Understanding [73.24847320536813]
  This study explores distilling visual information from pretrained multimodal transformers to pretrained language encoders. Our framework is inspired by cross-modal encoders' success in visual-language tasks while we alter the learning objective to cater to the language-heavy characteristics of NLU.
  arXiv  Detail & Related papers  (2022-04-15T03:44:00Z)
• Skeleton Based Sign Language Recognition Using Whole-body Keypoints [71.97020373520922]
  Sign language is used by deaf or speech impaired people to communicate. Skeleton-based recognition is becoming popular that it can be further ensembled with RGB-D based method to achieve state-of-the-art performance. Inspired by the recent development of whole-body pose estimation citejin 2020whole, we propose recognizing sign language based on the whole-body key points and features.
  arXiv  Detail & Related papers  (2021-03-16T03:38:17Z)
• Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision [110.66085917826648]
  We develop a technique that extrapolates multimodal alignments to language-only data by contextually mapping language tokens to their related images. "vokenization" is trained on relatively small image captioning datasets and we then apply it to generate vokens for large language corpora. Trained with these contextually generated vokens, our visually-supervised language models show consistent improvements over self-supervised alternatives on multiple pure-language tasks.
  arXiv  Detail & Related papers  (2020-10-14T02:11:51Z)
• Score-level Multi Cue Fusion for Sign Language Recognition [2.064612766965483]
  We propose a more straightforward approach to training cue models for Sign Language Recognition. We compare the performance of 3D Convolutional Neural Network (CNN) models specializing in dominant hand, hands, face, and upper body regions. Our experimental results have shown the effectiveness of mixed convolutional models.
  arXiv  Detail & Related papers  (2020-09-29T16:32:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.