Gloss-Free Representation Learning for Cross-Dataset Sign Spotting

• URL: http://arxiv.org/abs/2608.11332v1
• Date: Tue, 11 Aug 2026 18:31:57 GMT
• Title: Gloss-Free Representation Learning for Cross-Dataset Sign Spotting
• Abstract summary: We study whether weak transcript-based supervision can pretrain a reusable sign encoder in this setting.<n>We pretrain on TSL-News, a new Turkish broadcast corpus, using pseudo-gloss labels derived from transcripts.<n>We evaluate the learned representations via cross-dataset sign spotting on a new TSL Spotting Benchmark.
• Score: 0.8590590965724384
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Sign-language research for resource-constrained languages is often limited by the cost of dense linguistic labels such as glosses, temporal boundaries, and sign order. Broadcast news offers a practical alternative by pairing continuous signing with spoken-language transcripts, but this supervision is weak since text and signing are loosely aligned. Morphologically rich languages such as Turkish add further difficulty, as the same lexical meaning can appear in many inflected forms while some derived forms should remain distinct. We study whether weak transcript-based supervision can pretrain a reusable sign encoder in this setting, where poor text normalization can fragment pseudo-gloss targets and weaken representation learning. Unlike prior pseudo-gloss pipelines designed mainly to improve translation, we test whether the pretrained encoder transfers as a reusable representation for cross-dataset sign spotting. We pretrain on TSL-News, a new Turkish broadcast corpus, using pseudo-gloss labels derived from transcripts rather than manual annotation, comparing rule-based morphological lemmatization with constrained LLM-assisted normalization over a fixed vocabulary. We evaluate the learned representations via cross-dataset sign spotting on a new TSL Spotting Benchmark built from the TSL Dictionary corpus. The LLM-assisted encoder raises top-5 temporal localization mean IoU from 0.235 to 0.465, with 56.2% of examples reaching an IoU of at least 0.50; a frequency analysis suggests this gain is not mainly driven by memorizing frequent pseudo-gloss labels. In a downstream translation check, the same pretraining improves BLEU-4 from 9.60 to 11.04 and ROUGE from 23.48 to 27.43. These results show that loosely aligned broadcast data can provide effective weak supervision for learning sign representations that capture both lexical content and temporal structure.

Related papers

• Light-weight Pronunciation Assessment via Discrete Speech Token Surprisal [10.584132982823883]
  Training automated pronunciation assessment often relies on labeled learner errors or non-native corpora that are costly to collect.<n>We propose a lightweight framework trained only on native speech resources, operating unsupervised or lightly calibrated with a small set of scored utterances.
  arXiv  Detail & Related papers  (2026-06-18T08:04:16Z)
• Sign Spotting Disambiguation using Large Language Models [29.79050316749927]
  We introduce a training-free framework that integrates Large Language Models (LLMs) to significantly enhance sign spotting quality.<n>Our approach extracts global-temporal and hand shape features, which are then matched against a large-scale sign dictionary.<n>This dictionary-based matching inherently offers superior vocabulary flexibility without requiring model retraining.
  arXiv  Detail & Related papers  (2025-07-04T16:38:09Z)
• Bridging Sign and Spoken Languages: Pseudo Gloss Generation for Sign Language Translation [48.20483623444857]
  Sign Language Translation aims to map sign language videos to spoken language text.<n>A common approach relies on gloss annotations as an intermediate representation.<n>We propose a gloss-free pseudo gloss generation framework that eliminates the need for human-annotated glosses.
  arXiv  Detail & Related papers  (2025-05-21T12:19:55Z)
• Unlikelihood Tuning on Negative Samples Amazingly Improves Zero-Shot Translation [79.96416609433724]
  Zero-shot translation (ZST) aims to translate between unseen language pairs in training data. The common practice to guide the zero-shot language mapping during inference is to deliberately insert the source and target language IDs. Recent studies have shown that language IDs sometimes fail to navigate the ZST task, making them suffer from the off-target problem.
  arXiv  Detail & Related papers  (2023-09-28T17:02:36Z)
• Gloss-free Sign Language Translation: Improving from Visual-Language Pretraining [56.26550923909137]
  Gloss-Free Sign Language Translation (SLT) is a challenging task due to its cross-domain nature. We propose a novel Gloss-Free SLT based on Visual-Language Pretraining (GFSLT-) Our approach involves two stages: (i) integrating Contrastive Language-Image Pre-training with masked self-supervised learning to create pre-tasks that bridge the semantic gap between visual and textual representations and restore masked sentences, and (ii) constructing an end-to-end architecture with an encoder-decoder-like structure that inherits the parameters of the pre-trained Visual and Text Decoder from
  arXiv  Detail & Related papers  (2023-07-27T10:59:18Z)
• Keypoint based Sign Language Translation without Glosses [7.240731862549344]
  We propose a new keypoint normalization method for performing translation based on the skeleton point of the signer. It contributed to performance improvement by a customized normalization method depending on the body parts. Our method can be applied to various datasets in a way that can be applied to datasets without glosses.
  arXiv  Detail & Related papers  (2022-04-22T05:37:56Z)
• A Simple Multi-Modality Transfer Learning Baseline for Sign Language Translation [54.29679610921429]
  Existing sign language datasets contain only about 10K-20K pairs of sign videos, gloss annotations and texts. Data is thus a bottleneck for training effective sign language translation models. This simple baseline surpasses the previous state-of-the-art results on two sign language translation benchmarks.
  arXiv  Detail & Related papers  (2022-03-08T18:59:56Z)
• TSPNet: Hierarchical Feature Learning via Temporal Semantic Pyramid for Sign Language Translation [101.6042317204022]
  Sign language translation (SLT) aims to interpret sign video sequences into text-based natural language sentences. Existing SLT models usually represent sign visual features in a frame-wise manner. We develop a novel hierarchical sign video feature learning method via a temporal semantic pyramid network, called TSPNet.
  arXiv  Detail & Related papers  (2020-10-12T05:58:09Z)
• Leveraging Adversarial Training in Self-Learning for Cross-Lingual Text Classification [52.69730591919885]
  We present a semi-supervised adversarial training process that minimizes the maximal loss for label-preserving input perturbations. We observe significant gains in effectiveness on document and intent classification for a diverse set of languages.
  arXiv  Detail & Related papers  (2020-07-29T19:38:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.