HATL: Hierarchical Adaptive-Transfer Learning Framework for Sign Language Machine Translation

• URL: http://arxiv.org/abs/2603.19260v1
• Date: Thu, 26 Feb 2026 17:22:42 GMT
• Title: HATL: Hierarchical Adaptive-Transfer Learning Framework for Sign Language Machine Translation
• Abstract summary: Sign Language Machine Translation aims to bridge communication between Deaf and hearing individuals.<n>Existing transfer learning approaches in SLMT are static and often lead to overfitting.<n>We propose a Hierarchical Adaptive Transfer Learning framework, where pretrained layers are progressively and dynamically unfrozen.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Sign Language Machine Translation (SLMT) aims to bridge communication between Deaf and hearing individuals. However, its progress is constrained by scarce datasets, limited signer diversity, and large domain gaps between sign motion patterns and pretrained representations. Existing transfer learning approaches in SLMT are static and often lead to overfitting. These challenges call for the development of an adaptive framework that preserves pretrained structure while remaining robust across linguistic and signing variations. To fill this void, we propose a Hierarchical Adaptive Transfer Learning (HATL) framework, where pretrained layers are progressively and dynamically unfrozen based on training performance behavior. HATL combines dynamic unfreezing, layer-wise learning rate decay, and stability mechanisms to preserve generic representations while adapting to sign characteristics. We evaluate HATL on Sign2Text and Sign2Gloss2Text translation tasks using a pretrained ST-GCN++ backbone for feature extraction and the Transformer and an adaptive transformer (ADAT)for translation. To ensure robust multilingual generalization, we evaluate the proposed approach across three datasets: RWTH-PHOENIXWeather-2014 (PHOENIX14T), Isharah, and MedASL. Experimental results show that HATL consistently outperforms traditional transfer learning approaches across tasks and models, with ADAT achieving BLEU-4 improvements of 15.0% on PHOENIX14T and Isharah and 37.6% on MedASL.

Related papers

• Parameter-Efficient Adaptation of Pretrained Language Models for Time-Series Forecasting [2.141079906482723]
  We project fixed-length time-series patches directly into the embedding space of a pretrained GPT-2 backbone.<n>We analyze the effects of (i)representation strategy (continuous embeddings versus textual serialisation), (ii)adaptation regime (frozen backbone versus partial or full fine-tuning), and (iv)input context length.
  arXiv  Detail & Related papers  (2026-09-14T10:30:59Z)
• SCRIPT: Scalable Diffusion Policy with Multi-stage Training for Language-driven Physics-based Humanoid Control [72.5651722107621]
  SCRIPT is a scalable diffusion policy with a multi-stage training framework for language-driven physics-based humanoid control.<n>The core of SCRIPT is a Joint Action-State-Text Diffusion Transformer (JAST-DiT), which represents actions, physical states, and text as dedicated token streams.<n>To stabilize autore control, we introduce a nonlinear history conditioning mechanism, which preserves the dense recent context and samples increasingly sparse cues from long-term history.
  arXiv  Detail & Related papers  (2026-05-21T14:17:21Z)
• TextLDM: Language Modeling with Continuous Latent Diffusion [89.69255520673248]
  Diffusion Transformers (DiT) trained with flow matching in a VAE latent space have unified visual generation across images and videos.<n>We propose TextLDM, which transfers the visual latent diffusion recipe to text generation with minimal architectural modification.
  arXiv  Detail & Related papers  (2026-05-08T13:54:34Z)
• Gram-Anchored Prompt Learning for Vision-Language Models via Second-Order Statistics [6.327904555091201]
  We argue that relying solely on first-order information is insufficient for robust adaptation.<n>We propose textbfGram-Anchored Prompt Learning (GAPL) for Vision-Language Models via Second-Order Statistics.
  arXiv  Detail & Related papers  (2026-04-05T06:02:07Z)
• Learning Transferable Sensor Models via Language-Informed Pretraining [4.764303199378882]
  textbfSLIP is an open-source framework for learning language-aligned representations that generalize across diverse sensor setups.<n>SLIP integrates contrastive alignment with sensor-conditioned captioning, facilitating both discriminative understanding and generative reasoning.<n>Across 11 datasets, SLIP demonstrates superior performance in zero-shot transfer, signal captioning, and question answering.
  arXiv  Detail & Related papers  (2026-03-12T13:59:13Z)
• Improving Transfer Learning for Sequence Labeling Tasks by Adapting Pre-trained Neural Language Models [0.0]
  This thesis improves the transfer learning for sequence labeling tasks by adapting pre-trained neural language models.<n>The proposed improvements in transfer learning involve introducing a multi-task model that incorporates an additional signal.<n>The third improvement utilizes autoregressive large language models as text generators through a generative supervised in-context fine-tuning framework.
  arXiv  Detail & Related papers  (2025-10-22T21:23:53Z)
• MultiStream-LLM: Bridging Modalities for Robust Sign Language Translation [19.197388907510746]
  We introduce MultiStream-LLM, a modular framework designed to overcome limitations in automated sign language translation.<n>Our method establishes a new state-of-the-art on the How2Sign benchmark with a BLEU-4 score of 23.5 and 73.2% letter accuracy on the ChicagoFSWildPlus fingerspelling dataset.
  arXiv  Detail & Related papers  (2025-08-20T17:44:47Z)
• LANGTRAJ: Diffusion Model and Dataset for Language-Conditioned Trajectory Simulation [102.1527101235251]
  LangTraj is a language-conditioned scene-diffusion model that simulates the joint behavior of all agents in traffic scenarios.<n>By conditioning on natural language inputs, LangTraj provides flexible and intuitive control over interactive behaviors.<n>LangTraj demonstrates strong performance in realism, language controllability, and language-conditioned safety-critical simulation.
  arXiv  Detail & Related papers  (2025-04-15T17:14:06Z)
• Spatio-temporal transformer to support automatic sign language translation [0.0]
  This paper introduces a Transformer-based architecture that encodestemporal motion gestures, preserving both local and long-range spatial information.<n>The proposed approach was validated on the Colombian Sign Language Translation dataset.
  arXiv  Detail & Related papers  (2025-02-04T18:59:19Z)
• On the Transferability of Pre-trained Language Models: A Study from Artificial Datasets [74.11825654535895]
  Pre-training language models (LMs) on large-scale unlabeled text data makes the model much easier to achieve exceptional downstream performance. We study what specific traits in the pre-training data, other than the semantics, make a pre-trained LM superior to their counterparts trained from scratch on downstream tasks.
  arXiv  Detail & Related papers  (2021-09-08T10:39:57Z)
• Cross-lingual Transferring of Pre-trained Contextualized Language Models [73.97131976850424]
  We propose a novel cross-lingual model transferring framework for PrLMs: TreLM. To handle the symbol order and sequence length differences between languages, we propose an intermediate TRILayer" structure. We show the proposed framework significantly outperforms language models trained from scratch with limited data in both performance and efficiency.
  arXiv  Detail & Related papers  (2021-07-27T06:51:13Z)
• Unsupervised Paraphrasing with Pretrained Language Models [85.03373221588707]
  We propose a training pipeline that enables pre-trained language models to generate high-quality paraphrases in an unsupervised setting. Our recipe consists of task-adaptation, self-supervision, and a novel decoding algorithm named Dynamic Blocking. We show with automatic and human evaluations that our approach achieves state-of-the-art performance on both the Quora Question Pair and the ParaNMT datasets.
  arXiv  Detail & Related papers  (2020-10-24T11:55:28Z)
• Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation [59.38247587308604]
  We introduce a novel transformer based architecture that jointly learns Continuous Sign Language Recognition and Translation. We evaluate the recognition and translation performances of our approaches on the challenging RWTH-PHOENIX-Weather-2014T dataset. Our translation networks outperform both sign video to spoken language and gloss to spoken language translation models.
  arXiv  Detail & Related papers  (2020-03-30T21:35:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.