Hands-On: Segmenting Individual Signs from Continuous Sequences

• URL: http://arxiv.org/abs/2504.08593v2
• Date: Mon, 14 Apr 2025 08:07:48 GMT
• Title: Hands-On: Segmenting Individual Signs from Continuous Sequences
• Authors: Low Jian He, Harry Walsh, Ozge Mercanoglu Sincan, Richard Bowden,
• Abstract summary: We propose a transformer-based architecture that models the temporal dynamics of signing and frames segmentation.<n>Our model achieves state-of-the-art results on the DGS Corpus, while our features surpass prior benchmarks on BSLCorpus.
• Score: 28.01996053847279
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: This work tackles the challenge of continuous sign language segmentation, a key task with huge implications for sign language translation and data annotation. We propose a transformer-based architecture that models the temporal dynamics of signing and frames segmentation as a sequence labeling problem using the Begin-In-Out (BIO) tagging scheme. Our method leverages the HaMeR hand features, and is complemented with 3D Angles. Extensive experiments show that our model achieves state-of-the-art results on the DGS Corpus, while our features surpass prior benchmarks on BSLCorpus.

Related papers

• Signs as Tokens: A Retrieval-Enhanced Multilingual Sign Language Generator [55.94334001112357]
  We introduce a multilingual sign language model, Signs as Tokens (SOKE), which can generate 3D sign avatars autoregressively from text inputs. We propose a retrieval-enhanced SLG approach, which incorporates external sign dictionaries to provide accurate word-level signs.
  arXiv  Detail & Related papers  (2024-11-26T18:28:09Z)
• MS2SL: Multimodal Spoken Data-Driven Continuous Sign Language Production [93.32354378820648]
  We propose a unified framework for continuous sign language production, easing communication between sign and non-sign language users. A sequence diffusion model, utilizing embeddings extracted from text or speech, is crafted to generate sign predictions step by step. Experiments on How2Sign and PHOENIX14T datasets demonstrate that our model achieves competitive performance in sign language production.
  arXiv  Detail & Related papers  (2024-07-04T13:53:50Z)
• Sign Stitching: A Novel Approach to Sign Language Production [35.35777909051466]
  We propose using dictionary examples to create expressive sign language sequences. We present a 7-step approach to effectively stitch the signs together. We leverage the SignGAN model to map the output to a photo-realistic signer.
  arXiv  Detail & Related papers  (2024-05-13T11:44:57Z)
• Linguistically Motivated Sign Language Segmentation [51.06873383204105]
  We consider two kinds of segmentation: segmentation into individual signs and segmentation into phrases. Our method is motivated by linguistic cues observed in sign language corpora. We replace the predominant IO tagging scheme with BIO tagging to account for continuous signing.
  arXiv  Detail & Related papers  (2023-10-21T10:09:34Z)
• SignBERT+: Hand-model-aware Self-supervised Pre-training for Sign Language Understanding [132.78015553111234]
  Hand gesture serves as a crucial role during the expression of sign language. Current deep learning based methods for sign language understanding (SLU) are prone to over-fitting due to insufficient sign data resource. We propose the first self-supervised pre-trainable SignBERT+ framework with model-aware hand prior incorporated.
  arXiv  Detail & Related papers  (2023-05-08T17:16:38Z)
• BEST: BERT Pre-Training for Sign Language Recognition with Coupling Tokenization [135.73436686653315]
  We are dedicated to leveraging the BERT pre-training success and modeling the domain-specific statistics to fertilize the sign language recognition( SLR) model. Considering the dominance of hand and body in sign language expression, we organize them as pose triplet units and feed them into the Transformer backbone. Pre-training is performed via reconstructing the masked triplet unit from the corrupted input sequence. It adaptively extracts the discrete pseudo label from the pose triplet unit, which represents the semantic gesture/body state.
  arXiv  Detail & Related papers  (2023-02-10T06:23:44Z)
• Continuous 3D Multi-Channel Sign Language Production via Progressive Transformers and Mixture Density Networks [37.679114155300084]
  Sign Language Production (SLP) must embody both the continuous articulation and full morphology of sign to be truly understandable by the Deaf community. We propose a novel Progressive Transformer architecture, the first SLP model to translate from spoken language sentences to continuous 3D sign pose sequences. We present extensive data augmentation techniques to reduce prediction drift, alongside an adversarial training regime and a Mixture Density Network (MDN) formulation to produce realistic and expressive sign pose sequences.
  arXiv  Detail & Related papers  (2021-03-11T22:11:17Z)
• TSPNet: Hierarchical Feature Learning via Temporal Semantic Pyramid for Sign Language Translation [101.6042317204022]
  Sign language translation (SLT) aims to interpret sign video sequences into text-based natural language sentences. Existing SLT models usually represent sign visual features in a frame-wise manner. We develop a novel hierarchical sign video feature learning method via a temporal semantic pyramid network, called TSPNet.
  arXiv  Detail & Related papers  (2020-10-12T05:58:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.