Geo-Sign: Hyperbolic Contrastive Regularisation for Geometrically Aware Sign Language Translation

• URL: http://arxiv.org/abs/2506.00129v1
• Date: Fri, 30 May 2025 18:05:33 GMT
• Title: Geo-Sign: Hyperbolic Contrastive Regularisation for Geometrically Aware Sign Language Translation
• Authors: Edward Fish, Richard Bowden,
• Abstract summary: Geo-Sign is a method that leverages the properties of hyperbolic geometry to model the hierarchical structure inherent in sign language kinematics.<n>We introduce a hyperbolic projection layer, a weighted Fr'echet mean aggregation scheme, and a geometric contrastive loss operating directly in hyperbolic space.<n>These components are integrated into an end-to-end translation framework as a regularisation function, to enhance the representations within the language model.
• Score: 32.10033901054049
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Recent progress in Sign Language Translation (SLT) has focussed primarily on improving the representational capacity of large language models to incorporate Sign Language features. This work explores an alternative direction: enhancing the geometric properties of skeletal representations themselves. We propose Geo-Sign, a method that leverages the properties of hyperbolic geometry to model the hierarchical structure inherent in sign language kinematics. By projecting skeletal features derived from Spatio-Temporal Graph Convolutional Networks (ST-GCNs) into the Poincar\'e ball model, we aim to create more discriminative embeddings, particularly for fine-grained motions like finger articulations. We introduce a hyperbolic projection layer, a weighted Fr\'echet mean aggregation scheme, and a geometric contrastive loss operating directly in hyperbolic space. These components are integrated into an end-to-end translation framework as a regularisation function, to enhance the representations within the language model. This work demonstrates the potential of hyperbolic geometry to improve skeletal representations for Sign Language Translation, improving on SOTA RGB methods while preserving privacy and improving computational efficiency. Code available here: https://github.com/ed-fish/geo-sign.

Related papers

• StgcDiff: Spatial-Temporal Graph Condition Diffusion for Sign Language Transition Generation [33.695308849489784]
  We propose StgcDiff, a graph-based conditional diffusion framework that generates smooth transitions between discrete signs.<n>Specifically, we train an encoder-decoder architecture to learn a structure-aware representation of spatial-temporal skeleton.<n>We design the Sign-GCN module as the key component in our framework, which effectively models the spatial-temporal features.
  arXiv  Detail & Related papers  (2025-06-16T07:09:51Z)
• Graph Network for Sign Language Tasks [22.71156540352475]
  We introduce MixSignGraph, which represents sign sequences as a group of mixed graphs.<n>The LSG module learns the correlation of intra-frame cross-region features within one frame.<n>The TSG module tracks the interaction of inter-frame cross-region features among adjacent frames.<n>The HSG module aggregates the same-region features from different-granularity feature maps of a frame.
  arXiv  Detail & Related papers  (2025-04-16T12:23:30Z)
• Signs as Tokens: A Retrieval-Enhanced Multilingual Sign Language Generator [55.94334001112357]
  We introduce a multilingual sign language model, Signs as Tokens (SOKE), which can generate 3D sign avatars autoregressively from text inputs.<n>We propose a retrieval-enhanced SLG approach, which incorporates external sign dictionaries to provide accurate word-level signs.
  arXiv  Detail & Related papers  (2024-11-26T18:28:09Z)
• Linguistically Motivated Sign Language Segmentation [51.06873383204105]
  We consider two kinds of segmentation: segmentation into individual signs and segmentation into phrases. Our method is motivated by linguistic cues observed in sign language corpora. We replace the predominant IO tagging scheme with BIO tagging to account for continuous signing.
  arXiv  Detail & Related papers  (2023-10-21T10:09:34Z)
• Modeling Intensification for Sign Language Generation: A Computational Approach [13.57903290481737]
  End-to-end sign language generation models do not accurately represent the prosody in sign language. We aim to improve the prosody in generated sign languages by modeling intensification in a data-driven manner. We find that our efforts in intensification modeling yield better results when evaluated with automatic metrics.
  arXiv  Detail & Related papers  (2022-03-18T01:13:21Z)
• A Simple Multi-Modality Transfer Learning Baseline for Sign Language Translation [54.29679610921429]
  Existing sign language datasets contain only about 10K-20K pairs of sign videos, gloss annotations and texts. Data is thus a bottleneck for training effective sign language translation models. This simple baseline surpasses the previous state-of-the-art results on two sign language translation benchmarks.
  arXiv  Detail & Related papers  (2022-03-08T18:59:56Z)
• Skeletal Graph Self-Attention: Embedding a Skeleton Inductive Bias into Sign Language Production [37.679114155300084]
  Recent approaches to Sign Language Production (SLP) have adopted spoken language Neural Machine Translation (NMT) architectures, applied without sign-specific modifications. In this paper, we represent sign language sequences as a skeletal graph structure, with joints as nodes and both spatial and temporal connections as edges. We propose Skeletal Graph Self-Attention (SGSA), a novel graphical attention layer that embeds a skeleton bias into the SLP model.
  arXiv  Detail & Related papers  (2021-12-06T10:12:11Z)
• Sign Language Translation with Hierarchical Spatio-TemporalGraph Neural Network [6.623802929157273]
  Sign language translation (SLT) generates text in a spoken language from visual content in a sign language. In this paper, these unique characteristics of sign languages are formulated as hierarchical-temporal graph representations. A novel deep learning architecture, namely hierarchical hierarchical-temporal graph neural network (HSTG-NN), is proposed.
  arXiv  Detail & Related papers  (2021-11-14T07:02:28Z)
• Lattice-BERT: Leveraging Multi-Granularity Representations in Chinese Pre-trained Language Models [62.41139712595334]
  We propose a novel pre-training paradigm for Chinese -- Lattice-BERT. We construct a lattice graph from the characters and words in a sentence and feed all these text units into transformers. We show that our model can bring an average increase of 1.5% under the 12-layer setting.
  arXiv  Detail & Related papers  (2021-04-15T02:36:49Z)
• Spatial-spectral Hyperspectral Image Classification via Multiple Random Anchor Graphs Ensemble Learning [88.60285937702304]
  This paper proposes a novel spatial-spectral HSI classification method via multiple random anchor graphs ensemble learning (RAGE) Firstly, the local binary pattern is adopted to extract the more descriptive features on each selected band, which preserves local structures and subtle changes of a region. Secondly, the adaptive neighbors assignment is introduced in the construction of anchor graph, to reduce the computational complexity.
  arXiv  Detail & Related papers  (2021-03-25T09:31:41Z)
• TSPNet: Hierarchical Feature Learning via Temporal Semantic Pyramid for Sign Language Translation [101.6042317204022]
  Sign language translation (SLT) aims to interpret sign video sequences into text-based natural language sentences. Existing SLT models usually represent sign visual features in a frame-wise manner. We develop a novel hierarchical sign video feature learning method via a temporal semantic pyramid network, called TSPNet.
  arXiv  Detail & Related papers  (2020-10-12T05:58:09Z)
• Spatial Pyramid Based Graph Reasoning for Semantic Segmentation [67.47159595239798]
  We apply graph convolution into the semantic segmentation task and propose an improved Laplacian. The graph reasoning is directly performed in the original feature space organized as a spatial pyramid. We achieve comparable performance with advantages in computational and memory overhead.
  arXiv  Detail & Related papers  (2020-03-23T12:28:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.