Real-Time American Sign Language Recognition Using 3D Convolutional Neural Networks and LSTM: Architecture, Training, and Deployment

• URL: http://arxiv.org/abs/2512.22177v1
• Date: Fri, 19 Dec 2025 00:17:43 GMT
• Title: Real-Time American Sign Language Recognition Using 3D Convolutional Neural Networks and LSTM: Architecture, Training, and Deployment
• Authors: Dawnena Key,
• Abstract summary: This paper presents a real-time American Sign Language (ASL) recognition system utilizing a hybrid deep learning architecture.<n>The system processes webcam video streams to recognize word-level ASL signs, addressing communication barriers for over 70 million deaf and hard-of-hearing individuals worldwide.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper presents a real-time American Sign Language (ASL) recognition system utilizing a hybrid deep learning architecture combining 3D Convolutional Neural Networks (3D CNN) with Long Short-Term Memory (LSTM) networks. The system processes webcam video streams to recognize word-level ASL signs, addressing communication barriers for over 70 million deaf and hard-of-hearing individuals worldwide. Our architecture leverages 3D convolutions to capture spatial-temporal features from video frames, followed by LSTM layers that model sequential dependencies inherent in sign language gestures. Trained on the WLASL dataset (2,000 common words), ASL-LEX lexical database (~2,700 signs), and a curated set of 100 expert-annotated ASL signs, the system achieves F1-scores ranging from 0.71 to 0.99 across sign classes. The model is deployed on AWS infrastructure with edge deployment capability on OAK-D cameras for real-time inference. We discuss the architecture design, training methodology, evaluation metrics, and deployment considerations for practical accessibility applications.

Related papers

• MaDiS: Taming Masked Diffusion Language Models for Sign Language Generation [78.75809158246723]
  We present MaDiS, a masked-diffusion-based language model for SLG that captures bidirectional and supports efficient parallel multi-token generation.<n>We also introduce a tri-level cross-modal pretraining scheme that jointly learns from token-, latent-Hearing, and 3D-space objectives.<n>MaDiS achieves superior performance across multiple metrics, including DTW error and two newly introduced metrics, SiBLEU and SiCLIP, while reducing inference latency by nearly 30%.
  arXiv  Detail & Related papers  (2026-01-27T13:06:47Z)
• Lost in Translation, Found in Embeddings: Sign Language Translation and Alignment [84.39962912136525]
  We develop a model for sign language understanding that performs sign language translation (SLT) and sign-subtitle alignment (SSA)<n>Our approach is built upon three components: (i) a lightweight visual backbone that captures manual and non-manual cues from human keypoints and lip-region images; (ii) a Sliding Perceiver mapping network that aggregates consecutive visual features into word-level embeddings; and (iii) a multi-task scalable training strategy that jointly optimises SLT and SSA.
  arXiv  Detail & Related papers  (2025-12-08T21:05:46Z)
• Real-Time Sign Language to text Translation using Deep Learning: A Comparative study of LSTM and 3D CNN [0.0]
  This study investigates the performance of 3D Contemporalal Neural Networks (3D CNNs) and Long Short-Term Memory (LSTM) networks for real-time American Sign Language (ASL)<n> Experimental results demonstrate that 3D CNN achieve 92.4% recognition accuracy but require 3.2% more processing time per frame compared to LSTMs, which maintain 86.7% accuracy with significantly lower resource consumption.<n>This project provides professional benchmarks for developing assistive technologies, highlighting trade-offs between recognition precision and real-time operational requirements in edge computing environments.
  arXiv  Detail & Related papers  (2025-10-15T04:26:33Z)
• SHuBERT: Self-Supervised Sign Language Representation Learning via Multi-Stream Cluster Prediction [65.1590372072555]
  SHuBERT (Sign Hidden-Unit BERT) is a self-supervised contextual representation model learned from 1,000 hours of American Sign Language video.<n>SHuBERT adapts masked token prediction objectives to multi-stream visual sign language input, learning to predict multiple targets corresponding to clustered hand, face, and body pose streams.<n>SHuBERT achieves state-of-the-art performance across multiple tasks including sign language translation, isolated sign language recognition, and fingerspelling detection.
  arXiv  Detail & Related papers  (2024-11-25T03:13:08Z)
• Deep Neural Network-Based Sign Language Recognition: A Comprehensive Approach Using Transfer Learning with Explainability [0.0]
  We suggest a novel solution that uses a deep neural network to fully automate sign language recognition. This methodology integrates sophisticated preprocessing methodologies to optimise the overall performance. Our model's ability to provide informational clarity was assessed using the SHAP (SHapley Additive exPlanations) method.
  arXiv  Detail & Related papers  (2024-09-11T17:17:44Z)
• SCOPE: Sign Language Contextual Processing with Embedding from LLMs [49.5629738637893]
  Sign languages, used by around 70 million Deaf individuals globally, are visual languages that convey visual and contextual information. Current methods in vision-based sign language recognition ( SLR) and translation (SLT) struggle with dialogue scenes due to limited dataset diversity and the neglect of contextually relevant information. We introduce SCOPE, a novel context-aware vision-based SLR and SLT framework.
  arXiv  Detail & Related papers  (2024-09-02T08:56:12Z)
• Using CSNNs to Perform Event-based Data Processing & Classification on ASL-DVS [0.0]
  We develop a convolutional spiking neural network architecture to learn the spatial and temporal relations in the ASL-DVS gesture dataset. We performed classification on a pre-processed subset of the full ASL-DVS dataset to identify letter signs and achieved 100% training accuracy.
  arXiv  Detail & Related papers  (2024-08-01T14:49:43Z)
• Hierarchical I3D for Sign Spotting [39.69485385546803]
  We focus on the challenging of Sign Spotting instead of Isolated Sign Language Recognition. We propose a hierarchical sign spotting approach which learns coarse-to-finetemporal sign features. We achieve a state-of-the-art 0.607 F1 score, which was the top-1 winning solution of the ChaLearn 2022 Sign Spotting Challenge.
  arXiv  Detail & Related papers  (2022-10-03T14:07:23Z)
• Multi-View Spatial-Temporal Network for Continuous Sign Language Recognition [0.76146285961466]
  This paper proposes a multi-view spatial-temporal continuous sign language recognition network. It is tested on two public sign language datasets SLR-100 and PHOENIX-Weather 2014T (RWTH)
  arXiv  Detail & Related papers  (2022-04-19T08:43:03Z)
• Sign Language Recognition via Skeleton-Aware Multi-Model Ensemble [71.97020373520922]
  Sign language is commonly used by deaf or mute people to communicate. We propose a novel Multi-modal Framework with a Global Ensemble Model (GEM) for isolated Sign Language Recognition ( SLR) Our proposed SAM- SLR-v2 framework is exceedingly effective and achieves state-of-the-art performance with significant margins.
  arXiv  Detail & Related papers  (2021-10-12T16:57:18Z)
• Multi-Modal Zero-Shot Sign Language Recognition [51.07720650677784]
  We propose a multi-modal Zero-Shot Sign Language Recognition model. A Transformer-based model along with a C3D model is used for hand detection and deep features extraction. A semantic space is used to map the visual features to the lingual embedding of the class labels.
  arXiv  Detail & Related papers  (2021-09-02T09:10:39Z)
• Transferring Cross-domain Knowledge for Video Sign Language Recognition [103.9216648495958]
  Word-level sign language recognition (WSLR) is a fundamental task in sign language interpretation. We propose a novel method that learns domain-invariant visual concepts and fertilizes WSLR models by transferring knowledge of subtitled news sign to them.
  arXiv  Detail & Related papers  (2020-03-08T03:05:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.