Bootstrapping Sign Language Annotations with Sign Language Models

• URL: http://arxiv.org/abs/2604.07606v1
• Date: Wed, 08 Apr 2026 21:26:57 GMT
• Title: Bootstrapping Sign Language Annotations with Sign Language Models
• Abstract summary: AI-driven sign language interpretation is limited by a lack of high-quality annotated data.<n>New datasets contain professional interpreters and 100s of hours of data but remain only partially annotated and thus underutilized.<n>We develop a pseudo-annotation pipeline that takes signed video and English as input and outputs a ranked set of likely annotations.
• Score: 8.691348788683266
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: AI-driven sign language interpretation is limited by a lack of high-quality annotated data. New datasets including ASL STEM Wiki and FLEURS-ASL contain professional interpreters and 100s of hours of data but remain only partially annotated and thus underutilized, in part due to the prohibitive costs of annotating at this scale. In this work, we develop a pseudo-annotation pipeline that takes signed video and English as input and outputs a ranked set of likely annotations, including time intervals, for glosses, fingerspelled words, and sign classifiers. Our pipeline uses sparse predictions from our fingerspelling recognizer and isolated sign recognizer (ISR), along with a K-Shot LLM approach, to estimate these annotations. In service of this pipeline, we establish simple yet effective baseline fingerspelling and ISR models, achieving state-of-the-art on FSBoard (6.7% CER) and on ASL Citizen datasets (74% top-1 accuracy). To validate and provide a gold-standard benchmark, a professional interpreter annotated nearly 500 videos from ASL STEM Wiki with sequence-level gloss labels containing glosses, classifiers, and fingerspelling signs. These human annotations and over 300 hours of pseudo-annotations are being released in supplemental material.

Related papers

• SignSeek: Learning Transferable Representations for Sign Dictionary Retrieval [20.105725837533722]
  textbfSignSeek sets a new state-of-the-art performance in cross-corpus retrieval on sign languages.<n>It achieves zero-shot generalisation to an entirely unseen British Sign Language (BSL)
  arXiv  Detail & Related papers  (2026-09-03T11:31:47Z)
• Recognising BSL Fingerspelling in Continuous Signing Sequences [48.21890372724633]
  Fingerspelling is a critical component of British Sign Language (BSL)<n>Existing BSL fingerspelling datasets are either small in scale or temporally and letter-wise inaccurate.<n>We introduce a new large-scale BSL fingerspelling dataset, FS23K, constructed using an iterative annotation framework.
  arXiv  Detail & Related papers  (2026-03-19T23:29:31Z)
• Lost in Translation, Found in Embeddings: Sign Language Translation and Alignment [84.39962912136525]
  We develop a model for sign language understanding that performs sign language translation (SLT) and sign-subtitle alignment (SSA)<n>Our approach is built upon three components: (i) a lightweight visual backbone that captures manual and non-manual cues from human keypoints and lip-region images; (ii) a Sliding Perceiver mapping network that aggregates consecutive visual features into word-level embeddings; and (iii) a multi-task scalable training strategy that jointly optimises SLT and SSA.
  arXiv  Detail & Related papers  (2025-12-08T21:05:46Z)
• Seeing, Signing, and Saying: A Vision-Language Model-Assisted Pipeline for Sign Language Data Acquisition and Curation from Social Media [14.920401718793643]
  Vision Language Models (VLMs) have demonstrated strong capabilities as evaluators and real-time assistants.<n>We introduce the first automated annotation and filtering framework that utilizeVLMs to reduce reliance on manual effort while preserving data quality.<n>Our method is applied to TikTok videos across eight sign languages and to the already curated YouTube-SL-25 dataset in German Sign Language.
  arXiv  Detail & Related papers  (2025-10-29T11:29:56Z)
• SHuBERT: Self-Supervised Sign Language Representation Learning via Multi-Stream Cluster Prediction [65.1590372072555]
  SHuBERT (Sign Hidden-Unit BERT) is a self-supervised contextual representation model learned from 1,000 hours of American Sign Language video.<n>SHuBERT adapts masked token prediction objectives to multi-stream visual sign language input, learning to predict multiple targets corresponding to clustered hand, face, and body pose streams.<n>SHuBERT achieves state-of-the-art performance across multiple tasks including sign language translation, isolated sign language recognition, and fingerspelling detection.
  arXiv  Detail & Related papers  (2024-11-25T03:13:08Z)
• The Sem-Lex Benchmark: Modeling ASL Signs and Their Phonemes [6.0179345110920455]
  We introduce a new resource for American Sign Language (ASL) modeling, the Sem-Lex Benchmark. The Benchmark is the current largest of its kind, consisting of over 84k videos of isolated sign productions from deaf ASL signers who gave informed consent and received compensation. We present a suite of experiments which make use of the linguistic information in ASL-LEX, evaluating the practicality and fairness of the Sem-Lex Benchmark for isolated sign recognition (ISR)
  arXiv  Detail & Related papers  (2023-09-30T00:25:43Z)
• LSA-T: The first continuous Argentinian Sign Language dataset for Sign Language Translation [52.87578398308052]
  Sign language translation (SLT) is an active field of study that encompasses human-computer interaction, computer vision, natural language processing and machine learning. This paper presents the first continuous Argentinian Sign Language (LSA) dataset. It contains 14,880 sentence level videos of LSA extracted from the CN Sordos YouTube channel with labels and keypoints annotations for each signer.
  arXiv  Detail & Related papers  (2022-11-14T14:46:44Z)
• Automatic dense annotation of large-vocabulary sign language videos [85.61513254261523]
  We propose a simple, scalable framework to vastly increase the density of automatic annotations. We make these annotations publicly available to support the sign language research community.
  arXiv  Detail & Related papers  (2022-08-04T17:55:09Z)
• ASL-Homework-RGBD Dataset: An annotated dataset of 45 fluent and non-fluent signers performing American Sign Language homeworks [32.3809065803553]
  This dataset contains videos of fluent and non-fluent signers using American Sign Language (ASL) A total of 45 fluent and non-fluent participants were asked to perform signing homework assignments. The data is annotated to identify several aspects of signing including grammatical features and non-manual markers.
  arXiv  Detail & Related papers  (2022-07-08T17:18:49Z)
• BSL-1K: Scaling up co-articulated sign language recognition using mouthing cues [106.21067543021887]
  We show how to use mouthing cues from signers to obtain high-quality annotations from video data. The BSL-1K dataset is a collection of British Sign Language (BSL) signs of unprecedented scale.
  arXiv  Detail & Related papers  (2020-07-23T16:59:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.