SignMatch: Matching Dictionary Signs to Continuous Sign Language Video

• URL: http://arxiv.org/abs/2609.01886v1
• Date: Tue, 01 Sep 2026 21:33:33 GMT
• Title: SignMatch: Matching Dictionary Signs to Continuous Sign Language Video
• Abstract summary: We learn a prototype-structured sign embedding space from continuous video annotated with signs, where each learnable prototype corresponds to a sign class.<n> Isolated dictionary videos are then mapped into this sign space, enabling the matching between dictionary exemplars and continuous sign instances.
• Score: 61.094493488504845
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: The objective of this paper is to match dictionary sign videos to corresponding signs in continuous signing videos, where a match is defined by the visual similarity alone - the handshape and motion relative to the body. To achieve this, we learn a prototype-structured sign embedding space from continuous video annotated with signs, where each learnable prototype corresponds to a sign class. Isolated dictionary videos are then mapped into this sign space, enabling the matching between dictionary exemplars and continuous sign instances. This design supports direct dictionary-guided sign matching through embedding similarity and naturally extends to unseen signs using only dictionary exemplars. Experiments on ASL-Citizen dictionary retrieval, ChaLearn OSLWL dictionary-to-continuous sign matching, and using BOBSL's CSLR2 evaluation for automatic sign annotation demonstrate strong generalisation across datasets, tasks and sign languages. Without benchmark-specific supervision, the learned representation transfers effectively across American, British, and Spanish Sign Languages, outperforming prior methods on all three benchmarks. Project page: https://www.robots.ox.ac.uk/~vgg/research/signmatch/

Related papers

• SignSeek: Learning Transferable Representations for Sign Dictionary Retrieval [20.105725837533722]
  textbfSignSeek sets a new state-of-the-art performance in cross-corpus retrieval on sign languages.<n>It achieves zero-shot generalisation to an entirely unseen British Sign Language (BSL)
  arXiv  Detail & Related papers  (2026-09-03T11:31:47Z)
• New Capability to Look Up an ASL Sign from a Video Example [4.992008196032313]
  We describe a new system, publicly shared on the Web, to enable lookup of a video of an ASL sign. The user submits a video for analysis and is presented with the five most likely sign matches. This video lookup is also integrated into our newest version of SignStream software to facilitate linguistic annotation of ASL video data.
  arXiv  Detail & Related papers  (2024-07-18T15:14:35Z)
• Improving Continuous Sign Language Recognition with Cross-Lingual Signs [29.077175863743484]
  We study the feasibility of utilizing multilingual sign language corpora to facilitate continuous sign language recognition. We first build two sign language dictionaries containing isolated signs that appear in two datasets. Then we identify the sign-to-sign mappings between two sign languages via a well-optimized isolated sign language recognition model.
  arXiv  Detail & Related papers  (2023-08-21T15:58:47Z)
• Learnt Contrastive Concept Embeddings for Sign Recognition [33.72708697077754]
  We focus on explicitly creating sign embeddings that bridge the gap between sign language and spoken language. We train a vocabulary of embeddings that are based on the linguistic labels for sign video. We develop a conceptual similarity loss which is able to utilise word embeddings from NLP methods to create sign embeddings that have better sign language to spoken language correspondence.
  arXiv  Detail & Related papers  (2023-08-18T12:47:18Z)
• Weakly-supervised Fingerspelling Recognition in British Sign Language Videos [85.61513254261523]
  Previous fingerspelling recognition methods have not focused on British Sign Language (BSL) In contrast to previous methods, our method only uses weak annotations from subtitles for training. We propose a Transformer architecture adapted to this task, with a multiple-hypothesis CTC loss function to learn from alternative annotation possibilities.
  arXiv  Detail & Related papers  (2022-11-16T15:02:36Z)
• Scaling up sign spotting through sign language dictionaries [99.50956498009094]
  The focus of this work is $textitsign spotting$ - given a video of an isolated sign, our task is to identify $textitwhether$ and $textitwhere$ it has been signed in a continuous, co-articulated sign language video. We train a model using multiple types of available supervision by: (1) $textitwatching$ existing footage which is sparsely labelled using mouthing cues; (2) $textitreading$ associated subtitles which provide additional translations of the signed content. We validate the effectiveness of our approach on low
  arXiv  Detail & Related papers  (2022-05-09T10:00:03Z)
• Read and Attend: Temporal Localisation in Sign Language Videos [84.30262812057994]
  We train a Transformer model to ingest a continuous signing stream and output a sequence of written tokens. We show that it acquires the ability to attend to a large vocabulary of sign instances in the input sequence, enabling their localisation.
  arXiv  Detail & Related papers  (2021-03-30T16:39:53Z)
• Watch, read and lookup: learning to spot signs from multiple supervisors [99.50956498009094]
  Given a video of an isolated sign, our task is to identify whether and where it has been signed in a continuous, co-articulated sign language video. We train a model using multiple types of available supervision by: (1) watching existing sparsely labelled footage; (2) reading associated subtitles which provide additional weak-supervision; and (3) looking up words in visual sign language dictionaries. These three tasks are integrated into a unified learning framework using the principles of Noise Contrastive Estimation and Multiple Instance Learning.
  arXiv  Detail & Related papers  (2020-10-08T14:12:56Z)
• Transferring Cross-domain Knowledge for Video Sign Language Recognition [103.9216648495958]
  Word-level sign language recognition (WSLR) is a fundamental task in sign language interpretation. We propose a novel method that learns domain-invariant visual concepts and fertilizes WSLR models by transferring knowledge of subtitled news sign to them.
  arXiv  Detail & Related papers  (2020-03-08T03:05:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.