Fingerspelling Detection in American Sign Language
- URL: http://arxiv.org/abs/2104.01291v1
- Date: Sat, 3 Apr 2021 02:11:09 GMT
- Title: Fingerspelling Detection in American Sign Language
- Authors: Bowen Shi, Diane Brentari, Greg Shakhnarovich, Karen Livescu
- Abstract summary: We consider the task of fingerspelling detection in raw, untrimmed sign language videos.
This is an important step towards building real-world fingerspelling recognition systems.
We propose a benchmark and a suite of evaluation metrics, some of which reflect the effect of detection on the downstream fingerspelling recognition task.
- Score: 32.79935314131377
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Fingerspelling, in which words are signed letter by letter, is an important
component of American Sign Language. Most previous work on automatic
fingerspelling recognition has assumed that the boundaries of fingerspelling
regions in signing videos are known beforehand. In this paper, we consider the
task of fingerspelling detection in raw, untrimmed sign language videos. This
is an important step towards building real-world fingerspelling recognition
systems. We propose a benchmark and a suite of evaluation metrics, some of
which reflect the effect of detection on the downstream fingerspelling
recognition task. In addition, we propose a new model that learns to detect
fingerspelling via multi-task training, incorporating pose estimation and
fingerspelling recognition (transcription) along with detection, and compare
this model to several alternatives. The model outperforms all alternative
approaches across all metrics, establishing a state of the art on the
benchmark.
Related papers
- Sign Languague Recognition without frame-sequencing constraints: A proof
of concept on the Argentinian Sign Language [42.27617228521691]
This paper presents a general probabilistic model for sign classification that combines sub-classifiers based on different types of features.
The proposed model achieved an accuracy rate of 97% on an Argentinian Sign Language dataset.
arXiv Detail & Related papers (2023-10-26T14:47:11Z) - Toward American Sign Language Processing in the Real World: Data, Tasks,
and Methods [15.77894358993113]
I study automatic sign language processing in the wild, using signing videos collected from the Internet.
I present three new large-scale ASL datasets in the wild: ChicagoFSWild, ChicagoFSWild+, and OpenASL.
I propose two tasks for building real-world fingerspelling-based applications: fingerspelling detection and search.
arXiv Detail & Related papers (2023-08-23T20:38:19Z) - SignBERT+: Hand-model-aware Self-supervised Pre-training for Sign
Language Understanding [132.78015553111234]
Hand gesture serves as a crucial role during the expression of sign language.
Current deep learning based methods for sign language understanding (SLU) are prone to over-fitting due to insufficient sign data resource.
We propose the first self-supervised pre-trainable SignBERT+ framework with model-aware hand prior incorporated.
arXiv Detail & Related papers (2023-05-08T17:16:38Z) - On the Importance of Signer Overlap for Sign Language Detection [65.26091369630547]
We argue that the current benchmark data sets for sign language detection estimate overly positive results that do not generalize well.
We quantify this with a detailed analysis of the effect of signer overlap on current sign detection benchmark data sets.
We propose new data set partitions that are free of overlap and allow for more realistic performance assessment.
arXiv Detail & Related papers (2023-03-19T22:15:05Z) - Weakly-supervised Fingerspelling Recognition in British Sign Language
Videos [85.61513254261523]
Previous fingerspelling recognition methods have not focused on British Sign Language (BSL)
In contrast to previous methods, our method only uses weak annotations from subtitles for training.
We propose a Transformer architecture adapted to this task, with a multiple-hypothesis CTC loss function to learn from alternative annotation possibilities.
arXiv Detail & Related papers (2022-11-16T15:02:36Z) - Searching for fingerspelled content in American Sign Language [32.89182994277633]
Natural language processing for sign language video is crucial for making artificial intelligence technologies accessible to deaf individuals.
In this paper, we address the problem of searching for fingerspelled key-words or key phrases in raw sign language videos.
We propose an end-to-end model for this task, FSS-Net, that jointly detects fingerspelling and matches it to a text sequence.
arXiv Detail & Related papers (2022-03-24T18:36:22Z) - A Fine-Grained Visual Attention Approach for Fingerspelling Recognition
in the Wild [17.8181080354116]
Automatic recognition of fingerspelling can help resolve communication barriers when interacting with deaf people.
Main challenges prevalent in fingerspelling recognition are the ambiguity in the gestures and strong articulation of the hands.
We propose a fine-grained visual attention mechanism using the Transformer model for the sequence-to-sequence prediction task in the wild dataset.
arXiv Detail & Related papers (2021-05-17T06:15:35Z) - Skeleton Based Sign Language Recognition Using Whole-body Keypoints [71.97020373520922]
Sign language is used by deaf or speech impaired people to communicate.
Skeleton-based recognition is becoming popular that it can be further ensembled with RGB-D based method to achieve state-of-the-art performance.
Inspired by the recent development of whole-body pose estimation citejin 2020whole, we propose recognizing sign language based on the whole-body key points and features.
arXiv Detail & Related papers (2021-03-16T03:38:17Z) - Watch, read and lookup: learning to spot signs from multiple supervisors [99.50956498009094]
Given a video of an isolated sign, our task is to identify whether and where it has been signed in a continuous, co-articulated sign language video.
We train a model using multiple types of available supervision by: (1) watching existing sparsely labelled footage; (2) reading associated subtitles which provide additional weak-supervision; and (3) looking up words in visual sign language dictionaries.
These three tasks are integrated into a unified learning framework using the principles of Noise Contrastive Estimation and Multiple Instance Learning.
arXiv Detail & Related papers (2020-10-08T14:12:56Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.