SeRV: Semantic-Aligned Residual Vector Quantization for American Sign Language Generation

• URL: http://arxiv.org/abs/2609.05742v1
• Date: Fri, 04 Sep 2026 21:41:17 GMT
• Title: SeRV: Semantic-Aligned Residual Vector Quantization for American Sign Language Generation
• Abstract summary: SeRV is a semantic-aligned RVQ tokenizer for American Sign Language generation.<n>It learns a semantically structured residual token space by combining sentence-level motion-text alignment with token-level text-conditioned supervision.<n>It produces semantically consistent 3D ASL motion directly from text.
• Score: 16.011600536846434
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: American Sign Language (ASL) generation remains challenging due to limited paired text-ASL motion data and the difficulty of learning motion representations both precise for reconstruction and predictable from linguistic input. Existing methods rely on motion tokenizers optimized for reconstruction, without explicit semantic supervision from paired text. As a result, the learned tokens remain limited in supporting semantically consistent and fine-grained ASL motion generation. To address this limitation, we propose SeRV (Semantic-Aligned Residual Vector Quantization), a semantic-aligned RVQ tokenizer for ASL generation. SeRV learns a semantically structured residual token space by combining sentence-level motion-text alignment with token-level text-conditioned supervision. Building on this tokenizer, a Hierarchical GPT predicts residual motion tokens in a coarse-to-fine manner, generating structurally coherent and semantically aligned 3D ASL motion. We further construct a large-scale reconstructed 3D ASL motion-text benchmark by recovering paired 3D motion from YouTube-ASL videos. Experiments across 375 hours of ASL video show that SeRV achieves state-of-the-art pose accuracy on both How2Sign and YouTube-ASL datasets, while producing semantically consistent 3D ASL motion directly from text.

Related papers

• SignMimic: Robust High-Quality Sign Language Motion Generation via Human-Shape-Oblivious Pose Transfer Guidance [7.47809638392782]
  We study the challenge of sign language mimicking: given a driving and a single reference frame, synthesize a video where the target signer reproduces the source motion.<n>Prior pipelines entangle rigid motion, non-rigid deformation, and view-dependent completion in a monolithic generator.<n>We present Signimic, which (i) applies a TNet-based model to study SE(3) rigid canonicalization to stabilize global pose, (ii) performs non-rigid adaptation in a canonical space to preserve fine-grained articulators (hands/face) and coarticulation via NIF2D, and (iii)
  arXiv  Detail & Related papers  (2026-09-12T19:57:16Z)
• VTaMo: Video-Text Alignment Model for Sign Language Translation [5.958007180153739]
  We present VTaMo, a framework that introduces explicit multi-granularity alignment at three levels.<n> Experiments on Phoenix-2014T, CSL-Daily, How2Sign, and OpenASL demonstrate consistent state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-07-10T06:30:50Z)
• SignSparK: Efficient Multilingual Sign Language Production via Sparse Keyframe Learning [54.232148007248874]
  Current Sign Language Production (SLP) frameworks face a stark trade-off: direct text-to-pose models suffer from regression-to-the-mean effects.<n>We propose a novel training paradigm that leverages sparses to capture the true underlying distribution of human signing.<n>By predicting dense motion from these discrete anchors, our approach mitigates regression-to-the-mean while ensuring fluid articulation.
  arXiv  Detail & Related papers  (2026-03-11T06:02:36Z)
• MaDiS: Taming Masked Diffusion Language Models for Sign Language Generation [78.75809158246723]
  We present MaDiS, a masked-diffusion-based language model for SLG that captures bidirectional and supports efficient parallel multi-token generation.<n>We also introduce a tri-level cross-modal pretraining scheme that jointly learns from token-, latent-Hearing, and 3D-space objectives.<n>MaDiS achieves superior performance across multiple metrics, including DTW error and two newly introduced metrics, SiBLEU and SiCLIP, while reducing inference latency by nearly 30%.
  arXiv  Detail & Related papers  (2026-01-27T13:06:47Z)
• Lost in Translation, Found in Embeddings: Sign Language Translation and Alignment [84.39962912136525]
  We develop a model for sign language understanding that performs sign language translation (SLT) and sign-subtitle alignment (SSA)<n>Our approach is built upon three components: (i) a lightweight visual backbone that captures manual and non-manual cues from human keypoints and lip-region images; (ii) a Sliding Perceiver mapping network that aggregates consecutive visual features into word-level embeddings; and (iii) a multi-task scalable training strategy that jointly optimises SLT and SSA.
  arXiv  Detail & Related papers  (2025-12-08T21:05:46Z)
• Stable Signer: Hierarchical Sign Language Generative Model [26.707989581304734]
  We streamline the traditional redundant structure, simplify and optimize the task objective, and design a new sign language generative model called Stable Signer.<n>It redefines the SLP task as a hierarchical generation end-to-end task that only includes text understanding.<n>It generates hand gestures through the named SLP-MoE hand gesture rendering expert block to end-to-end generate high-quality and multi-style sign language videos.
  arXiv  Detail & Related papers  (2025-12-03T18:33:40Z)
• Advanced Sign Language Video Generation with Compressed and Quantized Multi-Condition Tokenization [20.063863466319326]
  SignViP is a novel framework that incorporates multiple fine-grained conditions for improved generation fidelity.<n>SignViP achieves state-of-the-art performance across metrics, including video quality temporal coherence, and semantic fidelity.
  arXiv  Detail & Related papers  (2025-06-19T02:56:06Z)
• GLOS: Sign Language Generation with Temporally Aligned Gloss-Level Conditioning [60.86278956347739]
  GLOS is a sign language generation framework with temporally aligned gloss-level conditioning.<n>Our method generates signs with correct lexical order and high semantic accuracy, outperforming prior methods on CSL-Daily and Phoenix-2014T.
  arXiv  Detail & Related papers  (2025-06-09T06:09:03Z)
• TSLFormer: A Lightweight Transformer Model for Turkish Sign Language Recognition Using Skeletal Landmarks [0.3749861135832072]
  TSLFormer treats sign gestures as ordered, string-like language.<n>Method only works with 3D joint positions extracted from Google's Mediapipe library.<n>Results show that joint-based input is sufficient for enabling real-time, mobile, and assistive communication systems for hearing-impaired individuals.
  arXiv  Detail & Related papers  (2025-05-11T14:30:56Z)
• Gloss-free Sign Language Translation: Improving from Visual-Language Pretraining [56.26550923909137]
  Gloss-Free Sign Language Translation (SLT) is a challenging task due to its cross-domain nature. We propose a novel Gloss-Free SLT based on Visual-Language Pretraining (GFSLT-) Our approach involves two stages: (i) integrating Contrastive Language-Image Pre-training with masked self-supervised learning to create pre-tasks that bridge the semantic gap between visual and textual representations and restore masked sentences, and (ii) constructing an end-to-end architecture with an encoder-decoder-like structure that inherits the parameters of the pre-trained Visual and Text Decoder from
  arXiv  Detail & Related papers  (2023-07-27T10:59:18Z)
• Video-based Sign Language Recognition without Temporal Segmentation [88.03159640595187]
  We propose a novel continuous sign recognition framework, which eliminates the preprocessing of temporal segmentation.<n>The proposed LS-HAN consists of three components: a two-stream Convolutional Neural Network (CNN) for video feature representation generation, a Latent Space for semantic gap bridging, and a Hierarchical Attention Network (HAN) for latent space based recognition.
  arXiv  Detail & Related papers  (2018-01-30T17:37:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.