Sign Language Conversation Interpretation Using Wearable Sensors and Machine Learning

• URL: http://arxiv.org/abs/2312.11903v1
• Date: Tue, 19 Dec 2023 07:06:32 GMT
• Title: Sign Language Conversation Interpretation Using Wearable Sensors and Machine Learning
• Authors: Basma Kalandar and Ziemowit Dworakowski
• Abstract summary: The count of people suffering from various levels of hearing loss reached 1.57 billion in 2019. This paper presents a proof of concept of an automatic sign language recognition system based on data obtained using a wearable device of 3 flex sensors.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The count of people suffering from various levels of hearing loss reached 1.57 billion in 2019. This huge number tends to suffer on many personal and professional levels and strictly needs to be included with the rest of society healthily. This paper presents a proof of concept of an automatic sign language recognition system based on data obtained using a wearable device of 3 flex sensors. The system is designed to interpret a selected set of American Sign Language (ASL) dynamic words by collecting data in sequences of the performed signs and using machine learning methods. The built models achieved high-quality performances, such as Random Forest with 99% accuracy, Support Vector Machine (SVM) with 99%, and two K-Nearest Neighbor (KNN) models with 98%. This indicates many possible paths toward the development of a full-scale system.

Related papers

• SHuBERT: Self-Supervised Sign Language Representation Learning via Multi-Stream Cluster Prediction [65.1590372072555]
  We introduce SHuBERT, a self-supervised transformer encoder that learns strong representations from American Sign Language (ASL) video content. Inspired by the success of the HuBERT speech representation model, SHuBERT adapts masked prediction for multi-stream visual sign language input. SHuBERT achieves state-of-the-art performance across multiple benchmarks.
  arXiv  Detail & Related papers  (2024-11-25T03:13:08Z)
• Sign language recognition based on deep learning and low-cost handcrafted descriptors [0.0]
  It is important to consider as many linguistic parameters as possible in gesture execution to avoid ambiguity between words. It is essential to ensure that the chosen technology is realistic, avoiding expensive, intrusive, or low-mobility sensors. We propose an efficient sign language recognition system that utilizes low-cost sensors and techniques.
  arXiv  Detail & Related papers  (2024-08-14T00:56:51Z)
• PenSLR: Persian end-to-end Sign Language Recognition Using Ensembling [0.953605234706973]
  Pen SLR is a glove-based sign language system consisting of an Inertial Measurement Unit (IMU) and five flexible sensors powered by a deep learning framework. We propose a novel ensembling technique by leveraging a multiple sequence alignment algorithm known as Star Alignment. Our evaluations show that Pen SLR achieves a remarkable word accuracy of 94.58% and 96.70% in subject-independent and subject-dependent setups.
  arXiv  Detail & Related papers  (2024-06-24T07:59:34Z)
• Sign Languague Recognition without frame-sequencing constraints: A proof of concept on the Argentinian Sign Language [42.27617228521691]
  This paper presents a general probabilistic model for sign classification that combines sub-classifiers based on different types of features. The proposed model achieved an accuracy rate of 97% on an Argentinian Sign Language dataset.
  arXiv  Detail & Related papers  (2023-10-26T14:47:11Z)
• Slovo: Russian Sign Language Dataset [83.93252084624997]
  This paper presents the Russian Sign Language (RSL) video dataset Slovo, produced using crowdsourcing platforms. The dataset contains 20,000 FullHD recordings, divided into 1,000 classes of isolated RSL gestures received by 194 signers.
  arXiv  Detail & Related papers  (2023-05-23T21:00:42Z)
• ASL Citizen: A Community-Sourced Dataset for Advancing Isolated Sign Language Recognition [6.296362537531586]
  Sign languages are used as a primary language by approximately 70 million D/deaf people world-wide. To help tackle this problem, we release ASL Citizen, the first crowdsourced Isolated Sign Language Recognition dataset. We propose that this dataset be used for sign language dictionary retrieval for American Sign Language (ASL), where a user demonstrates a sign to their webcam to retrieve matching signs from a dictionary.
  arXiv  Detail & Related papers  (2023-04-12T15:52:53Z)
• Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages [76.95115818308918]
  We introduce the Universal Speech Model (USM), a single large model that performs automatic speech recognition (ASR) across 100+ languages. This is achieved by pre-training the encoder of the model on a large unlabeled multilingual dataset of 12 million (M) hours spanning over 300 languages. We use multilingual pre-training with random-projection quantization and speech-text modality matching to achieve state-of-the-art performance on downstream multilingual ASR and speech-to-text translation tasks.
  arXiv  Detail & Related papers  (2023-03-02T07:47:18Z)
• From English to More Languages: Parameter-Efficient Model Reprogramming for Cross-Lingual Speech Recognition [50.93943755401025]
  We propose a new parameter-efficient learning framework based on neural model reprogramming for cross-lingual speech recognition. We design different auxiliary neural architectures focusing on learnable pre-trained feature enhancement. Our methods outperform existing ASR tuning architectures and their extension with self-supervised losses.
  arXiv  Detail & Related papers  (2023-01-19T02:37:56Z)
• Skeleton Based Sign Language Recognition Using Whole-body Keypoints [71.97020373520922]
  Sign language is used by deaf or speech impaired people to communicate. Skeleton-based recognition is becoming popular that it can be further ensembled with RGB-D based method to achieve state-of-the-art performance. Inspired by the recent development of whole-body pose estimation citejin 2020whole, we propose recognizing sign language based on the whole-body key points and features.
  arXiv  Detail & Related papers  (2021-03-16T03:38:17Z)
• Explicit Alignment Objectives for Multilingual Bidirectional Encoders [111.65322283420805]
  We present a new method for learning multilingual encoders, AMBER (Aligned Multilingual Bi-directional EncodeR) AMBER is trained on additional parallel data using two explicit alignment objectives that align the multilingual representations at different granularities. Experimental results show that AMBER obtains gains of up to 1.1 average F1 score on sequence tagging and up to 27.3 average accuracy on retrieval over the XLMR-large model.
  arXiv  Detail & Related papers  (2020-10-15T18:34:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.