DualAnchor: Preserving Language Priors and Improving Lexical Fidelity in Gloss-Free Sign Language Translation

• URL: http://arxiv.org/abs/2607.27614v1
• Date: Thu, 30 Jul 2026 03:02:13 GMT
• Title: DualAnchor: Preserving Language Priors and Improving Lexical Fidelity in Gloss-Free Sign Language Translation
• Abstract summary: We propose DualAnchor, a gloss-free sign language translation training framework.<n>It couples two complementary anchors for linguistically fluent and visually faithful generation.<n>TPA preserves the language prior by regularizing the multimodal decoder.<n>OTA improves lexical fidelity by formulating visual-textual matching as entropy-regularized partial optimal transport.
• Score: 30.38883468528891
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in large language models (LLMs) have led sign language translation (SLT), the task of converting sign-language videos into spoken-language text, to increasingly adopt LLMs as textual backbones. However, despite their strong language modeling capabilities, existing LLM-based SLT methods often undermine rather than exploit this language prior, producing disfluent translations, a failure we term language-prior degradation. Meanwhile, existing methods typically align videos and text at the sentence level, which does not ensure accurate lexical details and creates a lexical fidelity gap. To address both issues, we propose DualAnchor, a gloss-free LLM-based SLT training framework that couples two complementary anchors for linguistically fluent and visually faithful generation. Token-level Prior Anchoring (TPA) preserves the LLM's language prior by regularizing the multimodal decoder at each decoding step toward the next-token distribution of a frozen LLM conditioned on the same autoregressive prefix. Optimal Transport Alignment (OTA) improves lexical fidelity by formulating visual-textual matching as entropy-regularized partial optimal transport, with Sinkhorn optimization inducing a soft alignment between visual tokens and textual content tokens under a cosine cost. DualAnchor achieves strong overall performance on both PHOENIX-2014T and CSL-Daily. Targeted analyses attribute these gains to the complementary effects of the two anchors: TPA improves fluency, whereas OTA reduces fine-grained lexical errors.

Related papers

• RVLF: A Reinforcing Vision-Language Framework for Gloss-Free Sign Language Translation [44.39679803351263]
  We build a large vision-language model (LVLM) specifically designed for sign language.<n>For a sufficient representation of sign language, RVLF introduces an effective semantic representation learning mechanism.<n>Then, to improve sentence-level semantic misalignment, we introduce a GRPO-based optimization strategy.
  arXiv  Detail & Related papers  (2025-12-08T08:11:53Z)
• Languages are Modalities: Cross-Lingual Alignment via Encoder Injection [0.8461674097042394]
  We present a compute efficient language-as-modality method that conditions an instruction-tuned decoder without changing the tokenizer or retraining the decoder.<n>LLINK substantially improves bilingual retrieval and achieves 81.3% preference over the base model.<n>We find that improvements can be attributed to reduced tokenization inflation and a stronger cross lingual alignment.
  arXiv  Detail & Related papers  (2025-10-31T07:43:21Z)
• PART: Progressive Alignment Representation Training for Multilingual Speech-To-Text with LLMs [58.2469845374385]
  We introduce Progressive Alignment Representation Training (PART)<n>PART is a multi-stage and multi-task framework that separates within-language from cross-language alignment.<n>Experiments on CommonVoice 15, Fleurs, Wenetspeech, and CoVoST2 show that PART surpasses conventional approaches.
  arXiv  Detail & Related papers  (2025-09-24T03:54:14Z)
• Bridging Sign and Spoken Languages: Pseudo Gloss Generation for Sign Language Translation [48.20483623444857]
  Sign Language Translation aims to map sign language videos to spoken language text.<n>A common approach relies on gloss annotations as an intermediate representation.<n>We propose a gloss-free pseudo gloss generation framework that eliminates the need for human-annotated glosses.
  arXiv  Detail & Related papers  (2025-05-21T12:19:55Z)
• FlowDubber: Movie Dubbing with LLM-based Semantic-aware Learning and Flow Matching based Voice Enhancing [81.3306413498174]
  Movie Dubbing aims to convert scripts into speeches that align with the given movie clip in both temporal and emotional aspects.<n>Existing methods focus primarily on reducing the word error rate while ignoring the importance of lip-sync and acoustic quality.<n>We propose a large language model (LLM) based flow matching architecture for dubbing, named FlowDubber.
  arXiv  Detail & Related papers  (2025-05-02T13:30:19Z)
• LLaVA-SLT: Visual Language Tuning for Sign Language Translation [42.20090162339927]
  Recent advancements in Sign Language Translation (SLT) have shown promise, yet they often largely lag behind gloss-based approaches in terms of accuracy.<n>We introduce LLaVA-SLT, a pioneering Large Multimodal Model (LMM) framework designed to leverage the power of Large Language Models (LLMs) through effectively learned visual language embeddings.<n>Our comprehensive experiments demonstrate that LLaVA-SLT outperforms the state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-12-21T08:01:08Z)
• LLMs are Good Sign Language Translators [19.259163728870696]
  Sign Language Translation is a challenging task that aims to translate sign videos into spoken language. We propose a novel SignLLM framework to transform sign videos into a language-like representation. We achieve state-of-the-art gloss-free results on two widely-used SLT benchmarks.
  arXiv  Detail & Related papers  (2024-04-01T05:07:13Z)
• Building Accurate Translation-Tailored LLMs with Language Aware Instruction Tuning [57.323716555996114]
  Off-target translation remains an unsolved problem, especially for low-resource languages. Recent works have either designed advanced prompting strategies to highlight the functionality of translation instructions or exploited the in-context learning ability of LLMs. In this work, we design a two-stage fine-tuning algorithm to improve the instruction-following ability (especially the translation direction) of LLMs.
  arXiv  Detail & Related papers  (2024-03-21T13:47:40Z)
• Gloss-free Sign Language Translation: Improving from Visual-Language Pretraining [56.26550923909137]
  Gloss-Free Sign Language Translation (SLT) is a challenging task due to its cross-domain nature. We propose a novel Gloss-Free SLT based on Visual-Language Pretraining (GFSLT-) Our approach involves two stages: (i) integrating Contrastive Language-Image Pre-training with masked self-supervised learning to create pre-tasks that bridge the semantic gap between visual and textual representations and restore masked sentences, and (ii) constructing an end-to-end architecture with an encoder-decoder-like structure that inherits the parameters of the pre-trained Visual and Text Decoder from
  arXiv  Detail & Related papers  (2023-07-27T10:59:18Z)
• SimulSLT: End-to-End Simultaneous Sign Language Translation [55.54237194555432]
  Existing sign language translation methods need to read all the videos before starting the translation. We propose SimulSLT, the first end-to-end simultaneous sign language translation model. SimulSLT achieves BLEU scores that exceed the latest end-to-end non-simultaneous sign language translation model.
  arXiv  Detail & Related papers  (2021-12-08T11:04:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.