Seeing, Signing, and Saying: A Vision-Language Model-Assisted Pipeline for Sign Language Data Acquisition and Curation from Social Media

• URL: http://arxiv.org/abs/2510.25413v1
• Date: Wed, 29 Oct 2025 11:29:56 GMT
• Title: Seeing, Signing, and Saying: A Vision-Language Model-Assisted Pipeline for Sign Language Data Acquisition and Curation from Social Media
• Authors: Shakib Yazdani, Yasser Hamidullah, Cristina España-Bonet, Josef van Genabith,
• Abstract summary: Vision Language Models (VLMs) have demonstrated strong capabilities as evaluators and real-time assistants.<n>We introduce the first automated annotation and filtering framework that utilizeVLMs to reduce reliance on manual effort while preserving data quality.<n>Our method is applied to TikTok videos across eight sign languages and to the already curated YouTube-SL-25 dataset in German Sign Language.
• Score: 14.920401718793643
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Most existing sign language translation (SLT) datasets are limited in scale, lack multilingual coverage, and are costly to curate due to their reliance on expert annotation and controlled recording setup. Recently, Vision Language Models (VLMs) have demonstrated strong capabilities as evaluators and real-time assistants. Despite these advancements, their potential remains untapped in the context of sign language dataset acquisition. To bridge this gap, we introduce the first automated annotation and filtering framework that utilizes VLMs to reduce reliance on manual effort while preserving data quality. Our method is applied to TikTok videos across eight sign languages and to the already curated YouTube-SL-25 dataset in German Sign Language for the purpose of additional evaluation. Our VLM-based pipeline includes a face visibility detection, a sign activity recognition, a text extraction from video content, and a judgment step to validate alignment between video and text, implementing generic filtering, annotation and validation steps. Using the resulting corpus, TikTok-SL-8, we assess the performance of two off-the-shelf SLT models on our filtered dataset for German and American Sign Languages, with the goal of establishing baselines and evaluating the robustness of recent models on automatically extracted, slightly noisy data. Our work enables scalable, weakly supervised pretraining for SLT and facilitates data acquisition from social media.

Related papers

• Lost in Translation, Found in Embeddings: Sign Language Translation and Alignment [84.39962912136525]
  We develop a model for sign language understanding that performs sign language translation (SLT) and sign-subtitle alignment (SSA)<n>Our approach is built upon three components: (i) a lightweight visual backbone that captures manual and non-manual cues from human keypoints and lip-region images; (ii) a Sliding Perceiver mapping network that aggregates consecutive visual features into word-level embeddings; and (iii) a multi-task scalable training strategy that jointly optimises SLT and SSA.
  arXiv  Detail & Related papers  (2025-12-08T21:05:46Z)
• Logos as a Well-Tempered Pre-train for Sign Language Recognition [75.42794328290088]
  This paper presents Logos, a novel Russian Sign Language (RSL) dataset.<n>It is shown that a model, pre-trained on the Logos dataset can be used as a universal encoder for other language SLR tasks.<n>We show that explicitly labeling visually similar signs improves trained model quality as a visual encoder for downstream tasks.
  arXiv  Detail & Related papers  (2025-05-15T16:31:49Z)
• Lost in Translation, Found in Context: Sign Language Translation with Contextual Cues [56.038123093599815]
  Our objective is to translate continuous sign language into spoken language text.<n>We incorporate additional contextual cues together with the signing video.<n>We show that our contextual approach significantly enhances the quality of the translations.
  arXiv  Detail & Related papers  (2025-01-16T18:59:03Z)
• A Tale of Two Languages: Large-Vocabulary Continuous Sign Language Recognition from Spoken Language Supervision [74.972172804514]
  We introduce a multi-task Transformer model, CSLR2, that is able to ingest a signing sequence and output in a joint embedding space between signed language and spoken language text. New dataset annotations provide continuous sign-level annotations for six hours of test videos, and will be made publicly available. Our model significantly outperforms the previous state of the art on both tasks.
  arXiv  Detail & Related papers  (2024-05-16T17:19:06Z)
• Toward Informal Language Processing: Knowledge of Slang in Large Language Models [16.42982896928428]
  We construct a dataset that supports evaluation on a diverse set of tasks pertaining to automatic processing of slang. For both evaluation and finetuning, we show the effectiveness of our dataset on two core applications. We find that while LLMs such as GPT-4 achieve good performance in a zero-shot setting, smaller BERT-like models finetuned on our dataset achieve comparable performance.
  arXiv  Detail & Related papers  (2024-04-02T21:50:18Z)
• LSA-T: The first continuous Argentinian Sign Language dataset for Sign Language Translation [52.87578398308052]
  Sign language translation (SLT) is an active field of study that encompasses human-computer interaction, computer vision, natural language processing and machine learning. This paper presents the first continuous Argentinian Sign Language (LSA) dataset. It contains 14,880 sentence level videos of LSA extracted from the CN Sordos YouTube channel with labels and keypoints annotations for each signer.
  arXiv  Detail & Related papers  (2022-11-14T14:46:44Z)
• ASL-Homework-RGBD Dataset: An annotated dataset of 45 fluent and non-fluent signers performing American Sign Language homeworks [32.3809065803553]
  This dataset contains videos of fluent and non-fluent signers using American Sign Language (ASL) A total of 45 fluent and non-fluent participants were asked to perform signing homework assignments. The data is annotated to identify several aspects of signing including grammatical features and non-manual markers.
  arXiv  Detail & Related papers  (2022-07-08T17:18:49Z)
• WLASL-LEX: a Dataset for Recognising Phonological Properties in American Sign Language [2.814213966364155]
  We build a large-scale dataset of American Sign Language signs annotated with six different phonological properties. We investigate whether data-driven end-to-end and feature-based approaches can be optimised to automatically recognise these properties.
  arXiv  Detail & Related papers  (2022-03-11T17:21:24Z)
• A Simple Multi-Modality Transfer Learning Baseline for Sign Language Translation [54.29679610921429]
  Existing sign language datasets contain only about 10K-20K pairs of sign videos, gloss annotations and texts. Data is thus a bottleneck for training effective sign language translation models. This simple baseline surpasses the previous state-of-the-art results on two sign language translation benchmarks.
  arXiv  Detail & Related papers  (2022-03-08T18:59:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.