SignSeek: Learning Transferable Representations for Sign Dictionary Retrieval

• URL: http://arxiv.org/abs/2609.03695v1
• Date: Thu, 03 Sep 2026 11:31:47 GMT
• Title: SignSeek: Learning Transferable Representations for Sign Dictionary Retrieval
• Abstract summary: textbfSignSeek sets a new state-of-the-art performance in cross-corpus retrieval on sign languages.<n>It achieves zero-shot generalisation to an entirely unseen British Sign Language (BSL)
• Score: 20.105725837533722
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Sign language dictionaries are essential resources for sign language learners, yet automatically retrieving a sign from a dictionary, given only a query video, remains a challenging problem due to the natural variability between signers. Existing sign representation learning methods are built for closed-set recognition, producing embeddings that do not generalise to the open-set, signer-independent setting that retrieval demands. \textbf{SignSeek} closes this gap by contrastively learning sign representations with saliency-guided articulator masking. A contrastive objective aligns same-gloss signs across signers, while our Articulator Saliency-Guided Masking (ASGM) pinpoints the single most critical articulator per sign. This drives two complementary objectives, a masked contrastive alignment (MAC) loss that sees the sign through a single articulator and a masked prediction (MAP) loss that reconstructs it in latent space from the surrounding spatio-temporal context. Pretrained on 266K samples ($\sim$5,700 glosses) across multiple sign languages, \textbf{SignSeek} sets a new state-of-the-art performance in cross-corpus retrieval on ASL-Citizen, WLASL, and NMFs-CSL without any downstream fine-tuning. Strikingly, it achieves zero-shot generalisation to an entirely unseen British Sign Language (BSL), surpassing methods explicitly trained on BSL, and transfers seamlessly to isolated sign recognition and subtitle alignment, outperforming prior skeleton-based methods.

Related papers

• SignMatch: Matching Dictionary Signs to Continuous Sign Language Video [61.094493488504845]
  We learn a prototype-structured sign embedding space from continuous video annotated with signs, where each learnable prototype corresponds to a sign class.<n> Isolated dictionary videos are then mapped into this sign space, enabling the matching between dictionary exemplars and continuous sign instances.
  arXiv  Detail & Related papers  (2026-09-01T21:33:33Z)
• Bridging the Gap Between Semantics and Reconstruction:Unifying Sign Language Translation and Production [16.56134410439223]
  Recent advances in sign language research have shown a trend toward unifying multiple sign language understanding subtasks.<n>Can sign language understanding and production be unified within a single framework?<n>A unified framework must address two key challenges: bridging the gap between continuous sign motions and discrete text tokens.
  arXiv  Detail & Related papers  (2026-08-10T02:50:07Z)
• Recognising BSL Fingerspelling in Continuous Signing Sequences [48.21890372724633]
  Fingerspelling is a critical component of British Sign Language (BSL)<n>Existing BSL fingerspelling datasets are either small in scale or temporally and letter-wise inaccurate.<n>We introduce a new large-scale BSL fingerspelling dataset, FS23K, constructed using an iterative annotation framework.
  arXiv  Detail & Related papers  (2026-03-19T23:29:31Z)
• Pose-Based Sign Language Spotting via an End-to-End Encoder Architecture [0.4083182125683813]
  We present a first step toward sign language retrieval by addressing the challenge of detecting the presence or absence of a query sign video.<n>Unlike conventional approaches that rely on intermediate gloss recognition or text-based matching, we propose an end-to-end model that directly operates on pose keypoints extracted from sign videos.<n>Our architecture employs an encoder-only backbone with a binary classification head to determine whether the query sign appears within the target sequence.
  arXiv  Detail & Related papers  (2025-12-09T15:49:23Z)
• Sign Spotting Disambiguation using Large Language Models [29.79050316749927]
  We introduce a training-free framework that integrates Large Language Models (LLMs) to significantly enhance sign spotting quality.<n>Our approach extracts global-temporal and hand shape features, which are then matched against a large-scale sign dictionary.<n>This dictionary-based matching inherently offers superior vocabulary flexibility without requiring model retraining.
  arXiv  Detail & Related papers  (2025-07-04T16:38:09Z)
• Signs as Tokens: A Retrieval-Enhanced Multilingual Sign Language Generator [55.94334001112357]
  We introduce a multilingual sign language model, Signs as Tokens (SOKE), which can generate 3D sign avatars autoregressively from text inputs.<n>We propose a retrieval-enhanced SLG approach, which incorporates external sign dictionaries to provide accurate word-level signs.
  arXiv  Detail & Related papers  (2024-11-26T18:28:09Z)
• Self-Supervised Representation Learning with Spatial-Temporal Consistency for Sign Language Recognition [96.62264528407863]
  We propose a self-supervised contrastive learning framework to excavate rich context via spatial-temporal consistency. Inspired by the complementary property of motion and joint modalities, we first introduce first-order motion information into sign language modeling. Our method is evaluated with extensive experiments on four public benchmarks, and achieves new state-of-the-art performance with a notable margin.
  arXiv  Detail & Related papers  (2024-06-15T04:50:19Z)
• Improving Continuous Sign Language Recognition with Cross-Lingual Signs [29.077175863743484]
  We study the feasibility of utilizing multilingual sign language corpora to facilitate continuous sign language recognition. We first build two sign language dictionaries containing isolated signs that appear in two datasets. Then we identify the sign-to-sign mappings between two sign languages via a well-optimized isolated sign language recognition model.
  arXiv  Detail & Related papers  (2023-08-21T15:58:47Z)
• BSL-1K: Scaling up co-articulated sign language recognition using mouthing cues [106.21067543021887]
  We show how to use mouthing cues from signers to obtain high-quality annotations from video data. The BSL-1K dataset is a collection of British Sign Language (BSL) signs of unprecedented scale.
  arXiv  Detail & Related papers  (2020-07-23T16:59:01Z)
• Transferring Cross-domain Knowledge for Video Sign Language Recognition [103.9216648495958]
  Word-level sign language recognition (WSLR) is a fundamental task in sign language interpretation. We propose a novel method that learns domain-invariant visual concepts and fertilizes WSLR models by transferring knowledge of subtitled news sign to them.
  arXiv  Detail & Related papers  (2020-03-08T03:05:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.