Sign language recognition based on deep learning and low-cost handcrafted descriptors

• URL: http://arxiv.org/abs/2408.07244v1
• Date: Wed, 14 Aug 2024 00:56:51 GMT
• Title: Sign language recognition based on deep learning and low-cost handcrafted descriptors
• Authors: Alvaro Leandro Cavalcante Carneiro, Denis Henrique Pinheiro Salvadeo, Lucas de Brito Silva,
• Abstract summary: It is important to consider as many linguistic parameters as possible in gesture execution to avoid ambiguity between words. It is essential to ensure that the chosen technology is realistic, avoiding expensive, intrusive, or low-mobility sensors. We propose an efficient sign language recognition system that utilizes low-cost sensors and techniques.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In recent years, deep learning techniques have been used to develop sign language recognition systems, potentially serving as a communication tool for millions of hearing-impaired individuals worldwide. However, there are inherent challenges in creating such systems. Firstly, it is important to consider as many linguistic parameters as possible in gesture execution to avoid ambiguity between words. Moreover, to facilitate the real-world adoption of the created solution, it is essential to ensure that the chosen technology is realistic, avoiding expensive, intrusive, or low-mobility sensors, as well as very complex deep learning architectures that impose high computational requirements. Based on this, our work aims to propose an efficient sign language recognition system that utilizes low-cost sensors and techniques. To this end, an object detection model was trained specifically for detecting the interpreter's face and hands, ensuring focus on the most relevant regions of the image and generating inputs with higher semantic value for the classifier. Additionally, we introduced a novel approach to obtain features representing hand location and movement by leveraging spatial information derived from centroid positions of bounding boxes, thereby enhancing sign discrimination. The results demonstrate the efficiency of our handcrafted features, increasing accuracy by 7.96% on the AUTSL dataset, while adding fewer than 700 thousand parameters and incurring less than 10 milliseconds of additional inference time. These findings highlight the potential of our technique to strike a favorable balance between computational cost and accuracy, making it a promising approach for practical sign language recognition applications.

Related papers

• Deep Neural Network-Based Sign Language Recognition: A Comprehensive Approach Using Transfer Learning with Explainability [0.0]
  We suggest a novel solution that uses a deep neural network to fully automate sign language recognition. This methodology integrates sophisticated preprocessing methodologies to optimise the overall performance. Our model's ability to provide informational clarity was assessed using the SHAP (SHapley Additive exPlanations) method.
  arXiv  Detail & Related papers  (2024-09-11T17:17:44Z)
• Language-driven Grasp Detection with Mask-guided Attention [10.231956034184265]
  We propose a new method for language-driven grasp detection with mask-guided attention. Our approach integrates visual data, segmentation mask features, and natural language instructions. Our work introduces a new framework for language-driven grasp detection, paving the way for language-driven robotic applications.
  arXiv  Detail & Related papers  (2024-07-29T10:55:17Z)
• Sign Language Recognition Based On Facial Expression and Hand Skeleton [2.5879170041667523]
  We propose a sign language recognition network that integrates skeleton features of hands and facial expression. By incorporating facial expression information, the accuracy and robustness of sign language recognition are improved.
  arXiv  Detail & Related papers  (2024-07-02T13:02:51Z)
• PenSLR: Persian end-to-end Sign Language Recognition Using Ensembling [0.953605234706973]
  Pen SLR is a glove-based sign language system consisting of an Inertial Measurement Unit (IMU) and five flexible sensors powered by a deep learning framework. We propose a novel ensembling technique by leveraging a multiple sequence alignment algorithm known as Star Alignment. Our evaluations show that Pen SLR achieves a remarkable word accuracy of 94.58% and 96.70% in subject-independent and subject-dependent setups.
  arXiv  Detail & Related papers  (2024-06-24T07:59:34Z)
• Self-Supervised Representation Learning with Spatial-Temporal Consistency for Sign Language Recognition [96.62264528407863]
  We propose a self-supervised contrastive learning framework to excavate rich context via spatial-temporal consistency. Inspired by the complementary property of motion and joint modalities, we first introduce first-order motion information into sign language modeling. Our method is evaluated with extensive experiments on four public benchmarks, and achieves new state-of-the-art performance with a notable margin.
  arXiv  Detail & Related papers  (2024-06-15T04:50:19Z)
• Agile gesture recognition for capacitive sensing devices: adapting on-the-job [55.40855017016652]
  We demonstrate a hand gesture recognition system that uses signals from capacitive sensors embedded into the etee hand controller. The controller generates real-time signals from each of the wearer five fingers. We use a machine learning technique to analyse the time series signals and identify three features that can represent 5 fingers within 500 ms.
  arXiv  Detail & Related papers  (2023-05-12T17:24:02Z)
• On the Importance of Signer Overlap for Sign Language Detection [65.26091369630547]
  We argue that the current benchmark data sets for sign language detection estimate overly positive results that do not generalize well. We quantify this with a detailed analysis of the effect of signer overlap on current sign detection benchmark data sets. We propose new data set partitions that are free of overlap and allow for more realistic performance assessment.
  arXiv  Detail & Related papers  (2023-03-19T22:15:05Z)
• Multimodal Emotion Recognition using Transfer Learning from Speaker Recognition and BERT-based models [53.31917090073727]
  We propose a neural network-based emotion recognition framework that uses a late fusion of transfer-learned and fine-tuned models from speech and text modalities. We evaluate the effectiveness of our proposed multimodal approach on the interactive emotional dyadic motion capture dataset.
  arXiv  Detail & Related papers  (2022-02-16T00:23:42Z)
• Reinforced Iterative Knowledge Distillation for Cross-Lingual Named Entity Recognition [54.92161571089808]
  Cross-lingual NER transfers knowledge from rich-resource language to languages with low resources. Existing cross-lingual NER methods do not make good use of rich unlabeled data in target languages. We develop a novel approach based on the ideas of semi-supervised learning and reinforcement learning.
  arXiv  Detail & Related papers  (2021-06-01T05:46:22Z)
• Skeleton Based Sign Language Recognition Using Whole-body Keypoints [71.97020373520922]
  Sign language is used by deaf or speech impaired people to communicate. Skeleton-based recognition is becoming popular that it can be further ensembled with RGB-D based method to achieve state-of-the-art performance. Inspired by the recent development of whole-body pose estimation citejin 2020whole, we propose recognizing sign language based on the whole-body key points and features.
  arXiv  Detail & Related papers  (2021-03-16T03:38:17Z)
• Novel Approach to Use HU Moments with Image Processing Techniques for Real Time Sign Language Communication [0.0]
  "Sign Language Communicator" (SLC) is designed to solve the language barrier between the sign language users and the rest of the world. System is able to recognize selected Sign Language signs with the accuracy of 84% without a controlled background with small light adjustments.
  arXiv  Detail & Related papers  (2020-07-20T03:10:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.