ASL Recognition with Metric-Learning based Lightweight Network

• URL: http://arxiv.org/abs/2004.05054v1
• Date: Fri, 10 Apr 2020 14:41:30 GMT
• Title: ASL Recognition with Metric-Learning based Lightweight Network
• Authors: Evgeny Izutov
• Abstract summary: We propose a lightweight network for ASL gesture recognition with a performance sufficient for practical applications. The training code is available as part of Intel OpenVINO Training Extensions.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In the past decades the set of human tasks that are solved by machines was extended dramatically. From simple image classification problems researchers now move towards solving more sophisticated and vital problems, like, autonomous driving and language translation. The case of language translation includes a challenging area of sign language translation that incorporates both image and language processing. We make a step in that direction by proposing a lightweight network for ASL gesture recognition with a performance sufficient for practical applications. The proposed solution demonstrates impressive robustness on MS-ASL dataset and in live mode for continuous sign gesture recognition scenario. Additionally, we describe how to combine action recognition model training with metric-learning to train the network on the database of limited size. The training code is available as part of Intel OpenVINO Training Extensions.

Related papers

• Contrastive Localized Language-Image Pre-Training [60.4967533101887]
  Contrastive Language-Image Pre-training (CLIP) has been a celebrated method for training vision encoders to generate image/text representations. We propose Contrastive Localized Language-Image Pre-training (CLOC) by complementing CLIP with region-text contrastive loss and modules. CLOC enables high-quality regional embeddings for image region recognition and retrieval tasks.
  arXiv  Detail & Related papers  (2024-10-03T17:56:09Z)
• Deep Neural Network-Based Sign Language Recognition: A Comprehensive Approach Using Transfer Learning with Explainability [0.0]
  We suggest a novel solution that uses a deep neural network to fully automate sign language recognition. This methodology integrates sophisticated preprocessing methodologies to optimise the overall performance. Our model's ability to provide informational clarity was assessed using the SHAP (SHapley Additive exPlanations) method.
  arXiv  Detail & Related papers  (2024-09-11T17:17:44Z)
• ClawMachine: Fetching Visual Tokens as An Entity for Referring and Grounding [67.63933036920012]
  Existing methods, including proxy encoding and geometry encoding, incorporate additional syntax to encode the object's location. This study presents ClawMachine, offering a new methodology that notates an entity directly using the visual tokens. ClawMachine unifies visual referring and grounding into an auto-regressive format and learns with a decoder-only architecture.
  arXiv  Detail & Related papers  (2024-06-17T08:39:16Z)
• Multi-Stream Keypoint Attention Network for Sign Language Recognition and Translation [3.976851945232775]
  Current approaches for sign language recognition rely on RGB video inputs, which are vulnerable to fluctuations in the background. We propose a multi-stream keypoint attention network to depict a sequence of keypoints produced by a readily available keypoint estimator. We carry out comprehensive experiments on well-known benchmarks like Phoenix-2014, Phoenix-2014T, and CSL-Daily to showcase the efficacy of our methodology.
  arXiv  Detail & Related papers  (2024-05-09T10:58:37Z)
• Enhancing Brazilian Sign Language Recognition through Skeleton Image Representation [2.6311088262657907]
  This work proposes an Isolated Sign Language Recognition (ISLR) approach where body, hands, and facial landmarks are extracted throughout time and encoded as 2-D images. We show that our method surpassed the state-of-the-art in terms of performance metrics on two widely recognized datasets in Brazilian Sign Language (LIBRAS) In addition to being more accurate, our method is more time-efficient and easier to train due to its reliance on a simpler network architecture and solely RGB data as input.
  arXiv  Detail & Related papers  (2024-04-29T23:21:17Z)
• SignVTCL: Multi-Modal Continuous Sign Language Recognition Enhanced by Visual-Textual Contrastive Learning [51.800031281177105]
  SignVTCL is a continuous sign language recognition framework enhanced by visual-textual contrastive learning. It integrates multi-modal data (video, keypoints, and optical flow) simultaneously to train a unified visual backbone. It achieves state-of-the-art results compared with previous methods.
  arXiv  Detail & Related papers  (2024-01-22T11:04:55Z)
• Explore More Guidance: A Task-aware Instruction Network for Sign Language Translation Enhanced with Data Augmentation [20.125265661134964]
  Sign language recognition and translation first uses a recognition module to generate glosses from sign language videos. In this work, we propose a task-aware instruction network, namely TIN-SLT, for sign language translation.
  arXiv  Detail & Related papers  (2022-04-12T17:09:44Z)
• Reinforced Iterative Knowledge Distillation for Cross-Lingual Named Entity Recognition [54.92161571089808]
  Cross-lingual NER transfers knowledge from rich-resource language to languages with low resources. Existing cross-lingual NER methods do not make good use of rich unlabeled data in target languages. We develop a novel approach based on the ideas of semi-supervised learning and reinforcement learning.
  arXiv  Detail & Related papers  (2021-06-01T05:46:22Z)
• Skeleton Based Sign Language Recognition Using Whole-body Keypoints [71.97020373520922]
  Sign language is used by deaf or speech impaired people to communicate. Skeleton-based recognition is becoming popular that it can be further ensembled with RGB-D based method to achieve state-of-the-art performance. Inspired by the recent development of whole-body pose estimation citejin 2020whole, we propose recognizing sign language based on the whole-body key points and features.
  arXiv  Detail & Related papers  (2021-03-16T03:38:17Z)
• Application of Transfer Learning to Sign Language Recognition using an Inflated 3D Deep Convolutional Neural Network [0.0]
  Transfer learning is a technique to utilize a related task with an abundance of data available to help solve a target task lacking sufficient data. This paper investigates how effectively transfer learning can be applied to isolated sign language recognition.
  arXiv  Detail & Related papers  (2021-02-25T13:37:39Z)
• Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision [57.031588264841]
  We leverage a noisy dataset of over one billion image alt-text pairs, obtained without expensive filtering or post-processing steps. A simple dual-encoder architecture learns to align visual and language representations of the image and text pairs using a contrastive loss. We show that the scale of our corpus can make up for its noise and leads to state-of-the-art representations even with such a simple learning scheme.
  arXiv  Detail & Related papers  (2021-02-11T10:08:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.