On the Importance of Signer Overlap for Sign Language Detection
- URL: http://arxiv.org/abs/2303.10782v1
- Date: Sun, 19 Mar 2023 22:15:05 GMT
- Title: On the Importance of Signer Overlap for Sign Language Detection
- Authors: Abhilash Pal, Stephan Huber, Cyrine Chaabani, Alessandro Manzotti,
Oscar Koller
- Abstract summary: We argue that the current benchmark data sets for sign language detection estimate overly positive results that do not generalize well.
We quantify this with a detailed analysis of the effect of signer overlap on current sign detection benchmark data sets.
We propose new data set partitions that are free of overlap and allow for more realistic performance assessment.
- Score: 65.26091369630547
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Sign language detection, identifying if someone is signing or not, is
becoming crucially important for its applications in remote conferencing
software and for selecting useful sign data for training sign language
recognition or translation tasks. We argue that the current benchmark data sets
for sign language detection estimate overly positive results that do not
generalize well due to signer overlap between train and test partitions. We
quantify this with a detailed analysis of the effect of signer overlap on
current sign detection benchmark data sets. Comparing accuracy with and without
overlap on the DGS corpus and Signing in the Wild, we observed a relative
decrease in accuracy of 4.17% and 6.27%, respectively. Furthermore, we propose
new data set partitions that are free of overlap and allow for more realistic
performance assessment. We hope this work will contribute to improving the
accuracy and generalization of sign language detection systems.
Related papers
- signwriting-evaluation: Effective Sign Language Evaluation via SignWriting [3.484261625026626]
This paper introduces a comprehensive suite of evaluation metrics specifically designed for SignWriting.
We address the challenges of evaluating single signs versus continuous signing.
Our findings reveal the strengths and limitations of each metric, offering valuable insights for future advancements.
arXiv Detail & Related papers (2024-10-17T15:28:45Z) - EvSign: Sign Language Recognition and Translation with Streaming Events [59.51655336911345]
Event camera could naturally perceive dynamic hand movements, providing rich manual clues for sign language tasks.
We propose efficient transformer-based framework for event-based SLR and SLT tasks.
Our method performs favorably against existing state-of-the-art approaches with only 0.34% computational cost.
arXiv Detail & Related papers (2024-07-17T14:16:35Z) - Self-Supervised Representation Learning with Spatial-Temporal Consistency for Sign Language Recognition [96.62264528407863]
We propose a self-supervised contrastive learning framework to excavate rich context via spatial-temporal consistency.
Inspired by the complementary property of motion and joint modalities, we first introduce first-order motion information into sign language modeling.
Our method is evaluated with extensive experiments on four public benchmarks, and achieves new state-of-the-art performance with a notable margin.
arXiv Detail & Related papers (2024-06-15T04:50:19Z) - SignMusketeers: An Efficient Multi-Stream Approach for Sign Language Translation at Scale [22.49602248323602]
A persistent challenge in sign language video processing is how we learn representations of sign language.
Our proposed method focuses on just the most relevant parts in a signing video: the face, hands and body posture of the signer.
Our approach is based on learning from individual frames (rather than video sequences) and is therefore much more efficient than prior work on sign language pre-training.
arXiv Detail & Related papers (2024-06-11T03:00:41Z) - Improving Continuous Sign Language Recognition with Cross-Lingual Signs [29.077175863743484]
We study the feasibility of utilizing multilingual sign language corpora to facilitate continuous sign language recognition.
We first build two sign language dictionaries containing isolated signs that appear in two datasets.
Then we identify the sign-to-sign mappings between two sign languages via a well-optimized isolated sign language recognition model.
arXiv Detail & Related papers (2023-08-21T15:58:47Z) - Building Korean Sign Language Augmentation (KoSLA) Corpus with Data
Augmentation Technique [0.0]
We present an efficient framework of corpus for sign language translation.
By considering the linguistic features of sign language, our proposed framework is a first and unique attempt to build a multimodal sign language augmentation corpus.
arXiv Detail & Related papers (2022-07-12T02:12:36Z) - Keypoint based Sign Language Translation without Glosses [7.240731862549344]
We propose a new keypoint normalization method for performing translation based on the skeleton point of the signer.
It contributed to performance improvement by a customized normalization method depending on the body parts.
Our method can be applied to various datasets in a way that can be applied to datasets without glosses.
arXiv Detail & Related papers (2022-04-22T05:37:56Z) - Skeleton Based Sign Language Recognition Using Whole-body Keypoints [71.97020373520922]
Sign language is used by deaf or speech impaired people to communicate.
Skeleton-based recognition is becoming popular that it can be further ensembled with RGB-D based method to achieve state-of-the-art performance.
Inspired by the recent development of whole-body pose estimation citejin 2020whole, we propose recognizing sign language based on the whole-body key points and features.
arXiv Detail & Related papers (2021-03-16T03:38:17Z) - Watch, read and lookup: learning to spot signs from multiple supervisors [99.50956498009094]
Given a video of an isolated sign, our task is to identify whether and where it has been signed in a continuous, co-articulated sign language video.
We train a model using multiple types of available supervision by: (1) watching existing sparsely labelled footage; (2) reading associated subtitles which provide additional weak-supervision; and (3) looking up words in visual sign language dictionaries.
These three tasks are integrated into a unified learning framework using the principles of Noise Contrastive Estimation and Multiple Instance Learning.
arXiv Detail & Related papers (2020-10-08T14:12:56Z) - BSL-1K: Scaling up co-articulated sign language recognition using
mouthing cues [106.21067543021887]
We show how to use mouthing cues from signers to obtain high-quality annotations from video data.
The BSL-1K dataset is a collection of British Sign Language (BSL) signs of unprecedented scale.
arXiv Detail & Related papers (2020-07-23T16:59:01Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.