Stable Signer: Hierarchical Sign Language Generative Model

• URL: http://arxiv.org/abs/2512.04048v1
• Date: Wed, 03 Dec 2025 18:33:40 GMT
• Title: Stable Signer: Hierarchical Sign Language Generative Model
• Authors: Sen Fang, Yalin Feng, Hongbin Zhong, Yanxin Zhang, Dimitris N. Metaxas,
• Abstract summary: We streamline the traditional redundant structure, simplify and optimize the task objective, and design a new sign language generative model called Stable Signer.<n>It redefines the SLP task as a hierarchical generation end-to-end task that only includes text understanding.<n>It generates hand gestures through the named SLP-MoE hand gesture rendering expert block to end-to-end generate high-quality and multi-style sign language videos.
• Score: 26.707989581304734
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Sign Language Production (SLP) is the process of converting the complex input text into a real video. Most previous works focused on the Text2Gloss, Gloss2Pose, Pose2Vid stages, and some concentrated on Prompt2Gloss and Text2Avatar stages. However, this field has made slow progress due to the inaccuracy of text conversion, pose generation, and the rendering of poses into real human videos in these stages, resulting in gradually accumulating errors. Therefore, in this paper, we streamline the traditional redundant structure, simplify and optimize the task objective, and design a new sign language generative model called Stable Signer. It redefines the SLP task as a hierarchical generation end-to-end task that only includes text understanding (Prompt2Gloss, Text2Gloss) and Pose2Vid, and executes text understanding through our proposed new Sign Language Understanding Linker called SLUL, and generates hand gestures through the named SLP-MoE hand gesture rendering expert block to end-to-end generate high-quality and multi-style sign language videos. SLUL is trained using the newly developed Semantic-Aware Gloss Masking Loss (SAGM Loss). Its performance has improved by 48.6% compared to the current SOTA generation methods.

Related papers

• MaDiS: Taming Masked Diffusion Language Models for Sign Language Generation [78.75809158246723]
  We present MaDiS, a masked-diffusion-based language model for SLG that captures bidirectional and supports efficient parallel multi-token generation.<n>We also introduce a tri-level cross-modal pretraining scheme that jointly learns from token-, latent-Hearing, and 3D-space objectives.<n>MaDiS achieves superior performance across multiple metrics, including DTW error and two newly introduced metrics, SiBLEU and SiCLIP, while reducing inference latency by nearly 30%.
  arXiv  Detail & Related papers  (2026-01-27T13:06:47Z)
• Lost in Translation, Found in Embeddings: Sign Language Translation and Alignment [84.39962912136525]
  We develop a model for sign language understanding that performs sign language translation (SLT) and sign-subtitle alignment (SSA)<n>Our approach is built upon three components: (i) a lightweight visual backbone that captures manual and non-manual cues from human keypoints and lip-region images; (ii) a Sliding Perceiver mapping network that aggregates consecutive visual features into word-level embeddings; and (iii) a multi-task scalable training strategy that jointly optimises SLT and SSA.
  arXiv  Detail & Related papers  (2025-12-08T21:05:46Z)
• Mimir: Improving Video Diffusion Models for Precise Text Understanding [53.72393225042688]
  Text serves as the key control signal in video generation due to its narrative nature.<n>The recent success of large language models (LLMs) showcases the power of decoder-only transformers.<n>This work addresses this challenge with Mimir, an end-to-end training framework featuring a carefully tailored token fuser.
  arXiv  Detail & Related papers  (2024-12-04T07:26:44Z)
• A Spatio-Temporal Representation Learning as an Alternative to Traditional Glosses in Sign Language Translation and Production [9.065171626657818]
  This paper addresses the challenges associated with the use of glosses in Sign Language Translation (SLT) and Sign Language Production Language (SLP)<n>We introduce Universal Gloss-level Representation (UniGloR), a framework designed to capture thetemporal inherent sign language.<n>Our experiments in a keypoint-based setting demonstrate that UniGloR either outperforms or matches performance of previous SLT and SLP methods.
  arXiv  Detail & Related papers  (2024-07-03T07:12:36Z)
• PIXAR: Auto-Regressive Language Modeling in Pixel Space [51.530056034156374]
  We introduce PIXAR, the first pixel-based autoregressive LLM that performs text generation. Consisting of only a decoder, PIXAR can perform free-form generative tasks while keeping the number of parameters on par with previous encoder-decoder models. To overcome this problem, we propose an adversarial pretraining stage that improves the readability and accuracy of PIXAR by 8.1 on LAMBADA and 8.5 on bAbI.
  arXiv  Detail & Related papers  (2024-01-06T22:49:38Z)
• Sign Language Production with Latent Motion Transformer [2.184775414778289]
  We develop a new method to make high-quality sign videos without using human poses as a middle step. Our model works in two main parts: first, it learns from a generator and the video's hidden features, and next, it uses another model to understand the order of these hidden features. Compared with previous state-of-the-art approaches, our model performs consistently better on two word-level sign language datasets.
  arXiv  Detail & Related papers  (2023-12-20T10:53:06Z)
• Gloss-free Sign Language Translation: Improving from Visual-Language Pretraining [56.26550923909137]
  Gloss-Free Sign Language Translation (SLT) is a challenging task due to its cross-domain nature. We propose a novel Gloss-Free SLT based on Visual-Language Pretraining (GFSLT-) Our approach involves two stages: (i) integrating Contrastive Language-Image Pre-training with masked self-supervised learning to create pre-tasks that bridge the semantic gap between visual and textual representations and restore masked sentences, and (ii) constructing an end-to-end architecture with an encoder-decoder-like structure that inherits the parameters of the pre-trained Visual and Text Decoder from
  arXiv  Detail & Related papers  (2023-07-27T10:59:18Z)
• Cross-modality Data Augmentation for End-to-End Sign Language Translation [66.46877279084083]
  End-to-end sign language translation (SLT) aims to convert sign language videos into spoken language texts directly without intermediate representations. It has been a challenging task due to the modality gap between sign videos and texts and the data scarcity of labeled data. We propose a novel Cross-modality Data Augmentation (XmDA) framework to transfer the powerful gloss-to-text translation capabilities to end-to-end sign language translation.
  arXiv  Detail & Related papers  (2023-05-18T16:34:18Z)
• Being Comes from Not-being: Open-vocabulary Text-to-Motion Generation with Wordless Training [178.09150600453205]
  In this paper, we investigate offline open-vocabulary text-to-motion generation in a zero-shot learning manner. Inspired by the prompt learning in NLP, we pretrain a motion generator that learns to reconstruct the full motion from the masked motion. Our method reformulates the input text into a masked motion as the prompt for the motion generator to reconstruct'' the motion.
  arXiv  Detail & Related papers  (2022-10-28T06:20:55Z)
• Changing the Representation: Examining Language Representation for Neural Sign Language Production [43.45785951443149]
  We apply Natural Language Processing techniques to the first step of the Neural Sign Language Production pipeline. We use language models such as BERT and Word2Vec to create better sentence level embeddings. We introduce Text to HamNoSys (T2H) translation, and show the advantages of using a phonetic representation for sign language translation.
  arXiv  Detail & Related papers  (2022-09-16T12:45:29Z)
• SimulSLT: End-to-End Simultaneous Sign Language Translation [55.54237194555432]
  Existing sign language translation methods need to read all the videos before starting the translation. We propose SimulSLT, the first end-to-end simultaneous sign language translation model. SimulSLT achieves BLEU scores that exceed the latest end-to-end non-simultaneous sign language translation model.
  arXiv  Detail & Related papers  (2021-12-08T11:04:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.