New Capability to Look Up an ASL Sign from a Video Example

• URL: http://arxiv.org/abs/2407.13571v1
• Date: Thu, 18 Jul 2024 15:14:35 GMT
• Title: New Capability to Look Up an ASL Sign from a Video Example
• Authors: Carol Neidle, Augustine Opoku, Carey Ballard, Yang Zhou, Xiaoxiao He, Gregory Dimitriadis, Dimitris Metaxas,
• Abstract summary: We describe a new system, publicly shared on the Web, to enable lookup of a video of an ASL sign. The user submits a video for analysis and is presented with the five most likely sign matches. This video lookup is also integrated into our newest version of SignStream software to facilitate linguistic annotation of ASL video data.
• Score: 4.992008196032313
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Looking up an unknown sign in an ASL dictionary can be difficult. Most ASL dictionaries are organized based on English glosses, despite the fact that (1) there is no convention for assigning English-based glosses to ASL signs; and (2) there is no 1-1 correspondence between ASL signs and English words. Furthermore, what if the user does not know either the meaning of the target sign or its possible English translation(s)? Some ASL dictionaries enable searching through specification of articulatory properties, such as handshapes, locations, movement properties, etc. However, this is a cumbersome process and does not always result in successful lookup. Here we describe a new system, publicly shared on the Web, to enable lookup of a video of an ASL sign (e.g., a webcam recording or a clip from a continuous signing video). The user submits a video for analysis and is presented with the five most likely sign matches, in decreasing order of likelihood, so that the user can confirm the selection and then be taken to our ASLLRP Sign Bank entry for that sign. Furthermore, this video lookup is also integrated into our newest version of SignStream(R) software to facilitate linguistic annotation of ASL video data, enabling the user to directly look up a sign in the video being annotated, and, upon confirmation of the match, to directly enter into the annotation the gloss and features of that sign, greatly increasing the efficiency and consistency of linguistic annotations of ASL video data.

Related papers

• The American Sign Language Knowledge Graph: Infusing ASL Models with Linguistic Knowledge [6.481946043182915]
  We introduce the American Sign Language Knowledge Graph (ASLKG), compiled from twelve sources of expert linguistic knowledge. We use the ASLKG to train neuro-symbolic models for 3 ASL understanding tasks, achieving accuracies of 91% on ISR, 14% for predicting the semantic features of unseen signs, and 36% for classifying the topic of Youtube-ASL videos.
  arXiv  Detail & Related papers  (2024-11-06T00:16:16Z)
• C${^2}$RL: Content and Context Representation Learning for Gloss-free Sign Language Translation and Retrieval [37.12863427950066]
  We introduce an innovative pretraining paradigm for gloss-free SLRL, called C$2$RL. C$2$RL improves the BLEU-4 score by +5.3 on P14T, +10.6 on CSL-daily, +6.2 on OpenASL, and +1.3 on How2Sign. It also boosts the R@1 score by +8.3 on P14T, +14.4 on CSL-daily, and +5.9 on How2Sign.
  arXiv  Detail & Related papers  (2024-08-19T12:42:10Z)
• The Sem-Lex Benchmark: Modeling ASL Signs and Their Phonemes [6.0179345110920455]
  We introduce a new resource for American Sign Language (ASL) modeling, the Sem-Lex Benchmark. The Benchmark is the current largest of its kind, consisting of over 84k videos of isolated sign productions from deaf ASL signers who gave informed consent and received compensation. We present a suite of experiments which make use of the linguistic information in ASL-LEX, evaluating the practicality and fairness of the Sem-Lex Benchmark for isolated sign recognition (ISR)
  arXiv  Detail & Related papers  (2023-09-30T00:25:43Z)
• Gloss Attention for Gloss-free Sign Language Translation [60.633146518820325]
  We show how gloss annotations make sign language translation easier. We then propose emphgloss attention, which enables the model to keep its attention within video segments that have the same semantics locally. Experimental results on multiple large-scale sign language datasets show that our proposed GASLT model significantly outperforms existing methods.
  arXiv  Detail & Related papers  (2023-07-14T14:07:55Z)
• Automatic dense annotation of large-vocabulary sign language videos [85.61513254261523]
  We propose a simple, scalable framework to vastly increase the density of automatic annotations. We make these annotations publicly available to support the sign language research community.
  arXiv  Detail & Related papers  (2022-08-04T17:55:09Z)
• ASL-Homework-RGBD Dataset: An annotated dataset of 45 fluent and non-fluent signers performing American Sign Language homeworks [32.3809065803553]
  This dataset contains videos of fluent and non-fluent signers using American Sign Language (ASL) A total of 45 fluent and non-fluent participants were asked to perform signing homework assignments. The data is annotated to identify several aspects of signing including grammatical features and non-manual markers.
  arXiv  Detail & Related papers  (2022-07-08T17:18:49Z)
• Scaling up sign spotting through sign language dictionaries [99.50956498009094]
  The focus of this work is $textitsign spotting$ - given a video of an isolated sign, our task is to identify $textitwhether$ and $textitwhere$ it has been signed in a continuous, co-articulated sign language video. We train a model using multiple types of available supervision by: (1) $textitwatching$ existing footage which is sparsely labelled using mouthing cues; (2) $textitreading$ associated subtitles which provide additional translations of the signed content. We validate the effectiveness of our approach on low
  arXiv  Detail & Related papers  (2022-05-09T10:00:03Z)
• Watch, read and lookup: learning to spot signs from multiple supervisors [99.50956498009094]
  Given a video of an isolated sign, our task is to identify whether and where it has been signed in a continuous, co-articulated sign language video. We train a model using multiple types of available supervision by: (1) watching existing sparsely labelled footage; (2) reading associated subtitles which provide additional weak-supervision; and (3) looking up words in visual sign language dictionaries. These three tasks are integrated into a unified learning framework using the principles of Noise Contrastive Estimation and Multiple Instance Learning.
  arXiv  Detail & Related papers  (2020-10-08T14:12:56Z)
• BSL-1K: Scaling up co-articulated sign language recognition using mouthing cues [106.21067543021887]
  We show how to use mouthing cues from signers to obtain high-quality annotations from video data. The BSL-1K dataset is a collection of British Sign Language (BSL) signs of unprecedented scale.
  arXiv  Detail & Related papers  (2020-07-23T16:59:01Z)
• Recognizing American Sign Language Nonmanual Signal Grammar Errors in Continuous Videos [38.14850006590712]
  This paper introduces a near real-time system to recognize grammatical errors in continuous signing videos. Our system automatically recognizes if performance of ASL sentences contains grammatical errors made by ASL students.
  arXiv  Detail & Related papers  (2020-05-01T07:25:07Z)
• Transferring Cross-domain Knowledge for Video Sign Language Recognition [103.9216648495958]
  Word-level sign language recognition (WSLR) is a fundamental task in sign language interpretation. We propose a novel method that learns domain-invariant visual concepts and fertilizes WSLR models by transferring knowledge of subtitled news sign to them.
  arXiv  Detail & Related papers  (2020-03-08T03:05:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.