Hierarchical Feature Alignment for Gloss-Free Sign Language Translation

• URL: http://arxiv.org/abs/2507.06732v1
• Date: Wed, 09 Jul 2025 10:45:50 GMT
• Title: Hierarchical Feature Alignment for Gloss-Free Sign Language Translation
• Authors: Sobhan Asasi, Mohamed Ilyes Lakhal, Richard Bowden,
• Abstract summary: Sign Language Translation attempts to convert sign language videos into spoken sentences.<n>Existing methods struggle with disparity between visual and textual representations during end-to-end learning.<n>We introduce a novel hierarchical pre-training strategy inspired by the structure of sign language, incorporating pseudo-glosses and contrastive video-language alignment.
• Score: 29.544715933336715
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Sign Language Translation (SLT) attempts to convert sign language videos into spoken sentences. However, many existing methods struggle with the disparity between visual and textual representations during end-to-end learning. Gloss-based approaches help to bridge this gap by leveraging structured linguistic information. While, gloss-free methods offer greater flexibility and remove the burden of annotation, they require effective alignment strategies. Recent advances in Large Language Models (LLMs) have enabled gloss-free SLT by generating text-like representations from sign videos. In this work, we introduce a novel hierarchical pre-training strategy inspired by the structure of sign language, incorporating pseudo-glosses and contrastive video-language alignment. Our method hierarchically extracts features at frame, segment, and video levels, aligning them with pseudo-glosses and the spoken sentence to enhance translation quality. Experiments demonstrate that our approach improves BLEU-4 and ROUGE scores while maintaining efficiency.

Related papers

• Beyond Gloss: A Hand-Centric Framework for Gloss-Free Sign Language Translation [27.269988311306374]
  Sign Language Translation (SLT) is a challenging task that requires bridging the modality gap between visual and linguistic information.<n>We introduce textbfBeyondGloss, a novel gloss-free SLT framework that leverages thetemporal-aware reasoning capabilities of VideoLLMs.<n>Beyondtexttext achieves state-of-the-art performance on the PhoenixT14 and CSL-Daily benchmarks, demonstrating the effectiveness of the proposed framework.
  arXiv  Detail & Related papers  (2025-07-31T14:06:07Z)
• Bridging Sign and Spoken Languages: Pseudo Gloss Generation for Sign Language Translation [48.20483623444857]
  Sign Language Translation aims to map sign language videos to spoken language text.<n>A common approach relies on gloss annotations as an intermediate representation.<n>We propose a gloss-free pseudo gloss generation framework that eliminates the need for human-annotated glosses.
  arXiv  Detail & Related papers  (2025-05-21T12:19:55Z)
• An Efficient Sign Language Translation Using Spatial Configuration and Motion Dynamics with LLMs [7.630967411418269]
  Gloss-free Sign Language Translation (SLT) converts sign videos directly into spoken language sentences without relying on glosses.<n>This paper emphasizes the importance of capturing the spatial configurations and motion dynamics inherent in sign language.<n>We introduce Spatial and Motion-based Sign Language Translation (SpaMo), a novel LLM-based SLT framework.
  arXiv  Detail & Related papers  (2024-08-20T07:10:40Z)
• Gloss2Text: Sign Language Gloss translation using LLMs and Semantically Aware Label Smoothing [21.183453511034767]
  We propose several advances by leveraging pre-trained large language models (LLMs), data augmentation, and novel label-smoothing loss function. Our approach surpasses state-of-the-art performance in em Gloss2Text translation.
  arXiv  Detail & Related papers  (2024-07-01T15:46:45Z)
• Gloss-free Sign Language Translation: Improving from Visual-Language Pretraining [56.26550923909137]
  Gloss-Free Sign Language Translation (SLT) is a challenging task due to its cross-domain nature. We propose a novel Gloss-Free SLT based on Visual-Language Pretraining (GFSLT-) Our approach involves two stages: (i) integrating Contrastive Language-Image Pre-training with masked self-supervised learning to create pre-tasks that bridge the semantic gap between visual and textual representations and restore masked sentences, and (ii) constructing an end-to-end architecture with an encoder-decoder-like structure that inherits the parameters of the pre-trained Visual and Text Decoder from
  arXiv  Detail & Related papers  (2023-07-27T10:59:18Z)
• Gloss Attention for Gloss-free Sign Language Translation [60.633146518820325]
  We show how gloss annotations make sign language translation easier. We then propose emphgloss attention, which enables the model to keep its attention within video segments that have the same semantics locally. Experimental results on multiple large-scale sign language datasets show that our proposed GASLT model significantly outperforms existing methods.
  arXiv  Detail & Related papers  (2023-07-14T14:07:55Z)
• Gloss-Free End-to-End Sign Language Translation [59.28829048788345]
  We design the Gloss-Free End-to-end sign language translation framework (GloFE) Our method improves the performance of SLT in the gloss-free setting by exploiting the shared underlying semantics of signs and the corresponding spoken translation. We obtained state-of-the-art results on large-scale datasets, including OpenASL and How2Sign.
  arXiv  Detail & Related papers  (2023-05-22T09:57:43Z)
• Cross-modality Data Augmentation for End-to-End Sign Language Translation [66.46877279084083]
  End-to-end sign language translation (SLT) aims to convert sign language videos into spoken language texts directly without intermediate representations. It has been a challenging task due to the modality gap between sign videos and texts and the data scarcity of labeled data. We propose a novel Cross-modality Data Augmentation (XmDA) framework to transfer the powerful gloss-to-text translation capabilities to end-to-end sign language translation.
  arXiv  Detail & Related papers  (2023-05-18T16:34:18Z)
• GL-RG: Global-Local Representation Granularity for Video Captioning [52.56883051799501]
  We propose a GL-RG framework for video captioning, namely a textbfGlobal-textbfLocal textbfRepresentation textbfGranularity. Our GL-RG demonstrates three advantages over the prior efforts: 1) we explicitly exploit extensive visual representations from different video ranges to improve linguistic expression; 2) we devise a novel global-local encoder to produce rich semantic vocabulary to obtain a descriptive granularity of video contents across frames; and 3) we develop an incremental training strategy which organizes model learning in an incremental fashion to incur an optimal captioning
  arXiv  Detail & Related papers  (2022-05-22T02:00:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.