Continuous Sign Language Recognition System using Deep Learning with MediaPipe Holistic

• URL: http://arxiv.org/abs/2411.04517v1
• Date: Thu, 07 Nov 2024 08:19:39 GMT
• Title: Continuous Sign Language Recognition System using Deep Learning with MediaPipe Holistic
• Authors: Sharvani Srivastava, Sudhakar Singh, Pooja, Shiv Prakash,
• Abstract summary: Sign languages are the language of hearing-impaired people who use visuals for communication. Approximately 300 sign languages are being practiced worldwide such as American Sign Language (ASL), Chinese Sign Language (CSL), Indian Sign Language (ISL)
• Score: 1.9874264019909988
• License:
• Abstract: Sign languages are the language of hearing-impaired people who use visuals like the hand, facial, and body movements for communication. There are different signs and gestures representing alphabets, words, and phrases. Nowadays approximately 300 sign languages are being practiced worldwide such as American Sign Language (ASL), Chinese Sign Language (CSL), Indian Sign Language (ISL), and many more. Sign languages are dependent on the vocal language of a place. Unlike vocal or spoken languages, there are no helping words in sign language like is, am, are, was, were, will, be, etc. As only a limited population is well-versed in sign language, this lack of familiarity of sign language hinders hearing-impaired people from communicating freely and easily with everyone. This issue can be addressed by a sign language recognition (SLR) system which has the capability to translate the sign language into vocal language. In this paper, a continuous SLR system is proposed using a deep learning model employing Long Short-Term Memory (LSTM), trained and tested on an ISL primary dataset. This dataset is created using MediaPipe Holistic pipeline for tracking face, hand, and body movements and collecting landmarks. The system recognizes the signs and gestures in real-time with 88.23% accuracy.

Related papers

• Scaling up Multimodal Pre-training for Sign Language Understanding [96.17753464544604]
  Sign language serves as the primary meaning of communication for the deaf-mute community. To facilitate communication between the deaf-mute and hearing people, a series of sign language understanding (SLU) tasks have been studied. These tasks investigate sign language topics from diverse perspectives and raise challenges in learning effective representation of sign language videos.
  arXiv  Detail & Related papers  (2024-08-16T06:04:25Z)
• EvSign: Sign Language Recognition and Translation with Streaming Events [59.51655336911345]
  Event camera could naturally perceive dynamic hand movements, providing rich manual clues for sign language tasks. We propose efficient transformer-based framework for event-based SLR and SLT tasks. Our method performs favorably against existing state-of-the-art approaches with only 0.34% computational cost.
  arXiv  Detail & Related papers  (2024-07-17T14:16:35Z)
• Image-based Indian Sign Language Recognition: A Practical Review using Deep Neural Networks [0.0]
  This model is to develop a real-time word-level sign language recognition system that would translate sign language to text. For this analysis, the user must be able to take pictures of hand movements using a web camera. Our model is trained using a convolutional neural network (CNN), which is then utilized to recognize the images.
  arXiv  Detail & Related papers  (2023-04-28T09:27:04Z)
• Indian Sign Language Recognition Using Mediapipe Holistic [0.0]
  We will create a robust system for sign language recognition in order to convert Indian Sign Language to text or speech. The creation of a text-to-sign language paradigm is essential since it will enhance the sign language-dependent deaf and hard-of-hearing population's communication skills.
  arXiv  Detail & Related papers  (2023-04-20T12:25:47Z)
• All You Need In Sign Language Production [50.3955314892191]
  Sign language recognition and production need to cope with some critical challenges. We present an introduction to the Deaf culture, Deaf centers, psychological perspective of sign language. Also, the backbone architectures and methods in SLP are briefly introduced and the proposed taxonomy on SLP is presented.
  arXiv  Detail & Related papers  (2022-01-05T13:45:09Z)
• Sign Language Recognition System using TensorFlow Object Detection API [0.0]
  In this paper, we propose a method to create an Indian Sign Language dataset using a webcam and then using transfer learning, train a model to create a real-time Sign Language Recognition system. The system achieves a good level of accuracy even with a limited size dataset.
  arXiv  Detail & Related papers  (2022-01-05T07:13:03Z)
• Mandarin-English Code-switching Speech Recognition with Self-supervised Speech Representation Models [55.82292352607321]
  Code-switching (CS) is common in daily conversations where more than one language is used within a sentence. This paper uses the recently successful self-supervised learning (SSL) methods to leverage many unlabeled speech data without CS.
  arXiv  Detail & Related papers  (2021-10-07T14:43:35Z)
• Sign Language Production: A Review [51.07720650677784]
  Sign Language is the dominant yet non-primary form of communication language used in the deaf and hearing-impaired community. To make an easy and mutual communication between the hearing-impaired and the hearing communities, building a robust system capable of translating the spoken language into sign language is fundamental. To this end, sign language recognition and production are two necessary parts for making such a two-way system.
  arXiv  Detail & Related papers  (2021-03-29T19:38:22Z)
• Skeleton Based Sign Language Recognition Using Whole-body Keypoints [71.97020373520922]
  Sign language is used by deaf or speech impaired people to communicate. Skeleton-based recognition is becoming popular that it can be further ensembled with RGB-D based method to achieve state-of-the-art performance. Inspired by the recent development of whole-body pose estimation citejin 2020whole, we propose recognizing sign language based on the whole-body key points and features.
  arXiv  Detail & Related papers  (2021-03-16T03:38:17Z)
• Novel Approach to Use HU Moments with Image Processing Techniques for Real Time Sign Language Communication [0.0]
  "Sign Language Communicator" (SLC) is designed to solve the language barrier between the sign language users and the rest of the world. System is able to recognize selected Sign Language signs with the accuracy of 84% without a controlled background with small light adjustments.
  arXiv  Detail & Related papers  (2020-07-20T03:10:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.