Bridging Sign and Spoken Languages: Pseudo Gloss Generation for Sign Language Translation

• URL: http://arxiv.org/abs/2505.15438v1
• Date: Wed, 21 May 2025 12:19:55 GMT
• Title: Bridging Sign and Spoken Languages: Pseudo Gloss Generation for Sign Language Translation
• Authors: Jianyuan Guo, Peike Li, Trevor Cohn,
• Abstract summary: Sign Language Translation aims to map sign language videos to spoken language text.<n>A common approach relies on gloss annotations as an intermediate representation.<n>We propose a gloss-free pseudo gloss generation framework that eliminates the need for human-annotated glosses.
• Score: 48.20483623444857
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Sign Language Translation (SLT) aims to map sign language videos to spoken language text. A common approach relies on gloss annotations as an intermediate representation, decomposing SLT into two sub-tasks: video-to-gloss recognition and gloss-to-text translation. While effective, this paradigm depends on expert-annotated gloss labels, which are costly and rarely available in existing datasets, limiting its scalability. To address this challenge, we propose a gloss-free pseudo gloss generation framework that eliminates the need for human-annotated glosses while preserving the structured intermediate representation. Specifically, we prompt a Large Language Model (LLM) with a few example text-gloss pairs using in-context learning to produce draft sign glosses from spoken language text. To enhance the correspondence between LLM-generated pseudo glosses and the sign sequences in video, we correct the ordering in the pseudo glosses for better alignment via a weakly supervised learning process. This reordering facilitates the incorporation of auxiliary alignment objectives, and allows for the use of efficient supervision via a Connectionist Temporal Classification (CTC) loss. We train our SLT mode, which consists of a vision encoder and a translator, through a three-stage pipeline, which progressively narrows the modality gap between sign language and spoken language. Despite its simplicity, our approach outperforms previous state-of-the-art gloss-free frameworks on two SLT benchmarks and achieves competitive results compared to gloss-based methods.

Related papers

• Contrastive Pretraining with Dual Visual Encoders for Gloss-Free Sign Language Translation [33.48154010885497]
  Sign Language Translation (SLT) aims to convert sign language videos into spoken or written text.<n>We propose a two-phase, dual visual encoder framework for gloss-free SLT.
  arXiv  Detail & Related papers  (2025-07-14T14:09:36Z)
• Hierarchical Feature Alignment for Gloss-Free Sign Language Translation [29.544715933336715]
  Sign Language Translation attempts to convert sign language videos into spoken sentences.<n>Existing methods struggle with disparity between visual and textual representations during end-to-end learning.<n>We introduce a novel hierarchical pre-training strategy inspired by the structure of sign language, incorporating pseudo-glosses and contrastive video-language alignment.
  arXiv  Detail & Related papers  (2025-07-09T10:45:50Z)
• LLaVA-SLT: Visual Language Tuning for Sign Language Translation [42.20090162339927]
  Recent advancements in Sign Language Translation (SLT) have shown promise, yet they often largely lag behind gloss-based approaches in terms of accuracy.<n>We introduce LLaVA-SLT, a pioneering Large Multimodal Model (LMM) framework designed to leverage the power of Large Language Models (LLMs) through effectively learned visual language embeddings.<n>Our comprehensive experiments demonstrate that LLaVA-SLT outperforms the state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-12-21T08:01:08Z)
• A Spatio-Temporal Representation Learning as an Alternative to Traditional Glosses in Sign Language Translation and Production [9.065171626657818]
  This paper addresses the challenges associated with the use of glosses in Sign Language Translation (SLT) and Sign Language Production Language (SLP)<n>We introduce Universal Gloss-level Representation (UniGloR), a framework designed to capture thetemporal inherent sign language.<n>Our experiments in a keypoint-based setting demonstrate that UniGloR either outperforms or matches performance of previous SLT and SLP methods.
  arXiv  Detail & Related papers  (2024-07-03T07:12:36Z)
• A Tale of Two Languages: Large-Vocabulary Continuous Sign Language Recognition from Spoken Language Supervision [74.972172804514]
  We introduce a multi-task Transformer model, CSLR2, that is able to ingest a signing sequence and output in a joint embedding space between signed language and spoken language text. New dataset annotations provide continuous sign-level annotations for six hours of test videos, and will be made publicly available. Our model significantly outperforms the previous state of the art on both tasks.
  arXiv  Detail & Related papers  (2024-05-16T17:19:06Z)
• Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translation [30.008980708977095]
  We introduce Sign2GPT, a novel framework for sign language translation. We propose a novel pretraining strategy that directs our encoder to learn sign representations from automatically extracted pseudo-glosses. We evaluate our approach on two public benchmark sign language translation datasets.
  arXiv  Detail & Related papers  (2024-05-07T10:00:38Z)
• Gloss-free Sign Language Translation: Improving from Visual-Language Pretraining [56.26550923909137]
  Gloss-Free Sign Language Translation (SLT) is a challenging task due to its cross-domain nature. We propose a novel Gloss-Free SLT based on Visual-Language Pretraining (GFSLT-) Our approach involves two stages: (i) integrating Contrastive Language-Image Pre-training with masked self-supervised learning to create pre-tasks that bridge the semantic gap between visual and textual representations and restore masked sentences, and (ii) constructing an end-to-end architecture with an encoder-decoder-like structure that inherits the parameters of the pre-trained Visual and Text Decoder from
  arXiv  Detail & Related papers  (2023-07-27T10:59:18Z)
• Gloss Attention for Gloss-free Sign Language Translation [60.633146518820325]
  We show how gloss annotations make sign language translation easier. We then propose emphgloss attention, which enables the model to keep its attention within video segments that have the same semantics locally. Experimental results on multiple large-scale sign language datasets show that our proposed GASLT model significantly outperforms existing methods.
  arXiv  Detail & Related papers  (2023-07-14T14:07:55Z)
• Gloss-Free End-to-End Sign Language Translation [59.28829048788345]
  We design the Gloss-Free End-to-end sign language translation framework (GloFE) Our method improves the performance of SLT in the gloss-free setting by exploiting the shared underlying semantics of signs and the corresponding spoken translation. We obtained state-of-the-art results on large-scale datasets, including OpenASL and How2Sign.
  arXiv  Detail & Related papers  (2023-05-22T09:57:43Z)
• Cross-modality Data Augmentation for End-to-End Sign Language Translation [66.46877279084083]
  End-to-end sign language translation (SLT) aims to convert sign language videos into spoken language texts directly without intermediate representations. It has been a challenging task due to the modality gap between sign videos and texts and the data scarcity of labeled data. We propose a novel Cross-modality Data Augmentation (XmDA) framework to transfer the powerful gloss-to-text translation capabilities to end-to-end sign language translation.
  arXiv  Detail & Related papers  (2023-05-18T16:34:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.