ViPo-MLLM: Visual-Pose Multimodal LLM for Gloss-Free Sign Language Translation

• URL: http://arxiv.org/abs/2607.03657v1
• Date: Sat, 04 Jul 2026 01:17:48 GMT
• Title: ViPo-MLLM: Visual-Pose Multimodal LLM for Gloss-Free Sign Language Translation
• Abstract summary: Existing methods often use single-modality or weakly fused features, limiting performance.<n>We propose Vi-Po-MLLM, a framework that integrates intra-temporal and dynamics cross-modal attention.<n>The proposed model was evaluated on the ENIXENIX14T and CSL-Daily datasets and achieved new state-the-art results.
• Score: 23.358041920240208
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Gloss-free Sign Language Translation (SLT) translates sign language videos into spoken-language sentences without gloss annotations, avoiding costly labeling but requiring fine-grained modeling of hands, body, and facial cues. Existing methods often use single-modality or weakly fused features, limiting performance. We propose ViPo-MLLM, a framework that integrates spatio-temporal RGB and human pose features. Dedicated encoders model intra-modal dynamics and cross-modal attention captures long-range dependencies. The fused representation is conditioned with a structured prompt and processed by an LLM trained with contrastive and language modeling objectives. The proposed model was evaluated on the PHOENIX14T and CSL-Daily datasets and achieved new state-of-the-art results on both datasets. Moreover, the ViPo-MLLM model attained competitive performance compared to gloss-based recognition approaches, confirming the effectiveness of the proposed pose cues and cross-modal attention mechanisms.

Related papers

• Seeing Beyond Words: Self-Supervised Visual Learning for Multimodal Large Language Models [53.06230963851451]
  JARVIS is a JEPA-inspired framework for self-supervised visual enhancement in MLLMs.<n>We introduce JARVIS, a JEPA-inspired framework for self-supervised visual enhancement in MLLMs.
  arXiv  Detail & Related papers  (2025-12-17T19:01:34Z)
• Beyond Gloss: A Hand-Centric Framework for Gloss-Free Sign Language Translation [18.544615425772903]
  Sign Language Translation (SLT) is a challenging task that requires bridging the modality gap between visual and linguistic information.<n>We introduce textbfBeyondGloss, a novel gloss-free SLT framework that leverages thetemporal-aware reasoning capabilities of VideoLLMs.<n>Beyondtexttext achieves state-of-the-art performance on the PhoenixT14 and CSL-Daily benchmarks, demonstrating the effectiveness of the proposed framework.
  arXiv  Detail & Related papers  (2025-07-31T14:06:07Z)
• ByDeWay: Boost Your multimodal LLM with DEpth prompting in a Training-Free Way [8.242020781632801]
  ByDeWay is a training-free framework designed to enhance the performance of Multimodal Large Language Models (MLLMs)<n>ByDeWay uses a novel prompting strategy called Layered-Depth-Based Prompting (LDP)<n>It segments the scene into closest, mid-range, and farthest layers using monocular depth estimation, then generates region-specific captions with a grounded vision-language model.
  arXiv  Detail & Related papers  (2025-07-11T15:21:49Z)
• How Visual Representations Map to Language Feature Space in Multimodal LLMs [9.880509106657009]
  We study the mechanism by which vision-language models (VLMs) achieve alignment of visual and linguistic representations.<n>By keeping the language model frozen, we ensure it maintains its original language representations without adaptation to visual data.<n>We reveal the layer-wise progression through which visual representations gradually align with language feature representations, converging in middle-to-later layers.
  arXiv  Detail & Related papers  (2025-06-13T17:34:05Z)
• HoVLE: Unleashing the Power of Monolithic Vision-Language Models with Holistic Vision-Language Embedding [91.0552157725366]
  This paper presents a novel high-performance monolithic VLM named HoVLE.<n>It converts visual and textual inputs into a shared space, allowing LLMs to process images in the same way as texts.<n>Our experiments show that HoVLE achieves performance close to leading compositional models on various benchmarks.
  arXiv  Detail & Related papers  (2024-12-20T18:59:59Z)
• Leveraging the Power of MLLMs for Gloss-Free Sign Language Translation [14.817951264354022]
  Sign language translation (SLT) is a challenging task that involves translating sign language images into spoken language.<n>We propose a novel gloss-free framework called Multimodal Sign Language Translation (MMSLT)<n>Our approach achieves state-of-the-art performance on benchmark datasets PHOENIX14T and CSL-Daily.
  arXiv  Detail & Related papers  (2024-11-25T09:01:41Z)
• Towards Semantic Equivalence of Tokenization in Multimodal LLM [149.11720372278273]
  Vision tokenization is essential for semantic alignment between vision and language.<n>This paper proposes a novel dynamic Semantic-Equivalent Vision Tokenizer (SeTok)<n>SeTok groups visual features into semantic units via a dynamic clustering algorithm.<n>The resulting vision tokens effectively preserve semantic integrity and capture both low-frequency and high-frequency visual features.
  arXiv  Detail & Related papers  (2024-06-07T17:55:43Z)
• Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception [63.03288425612792]
  We propose bfAnyRef, a general MLLM model that can generate pixel-wise object perceptions and natural language descriptions from multi-modality references. Our model achieves state-of-the-art results across multiple benchmarks, including diverse modality referring segmentation and region-level referring expression generation.
  arXiv  Detail & Related papers  (2024-03-05T13:45:46Z)
• SignVTCL: Multi-Modal Continuous Sign Language Recognition Enhanced by Visual-Textual Contrastive Learning [51.800031281177105]
  SignVTCL is a continuous sign language recognition framework enhanced by visual-textual contrastive learning. It integrates multi-modal data (video, keypoints, and optical flow) simultaneously to train a unified visual backbone. It achieves state-of-the-art results compared with previous methods.
  arXiv  Detail & Related papers  (2024-01-22T11:04:55Z)
• Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization [52.935150075484074]
  We introduce a well-designed visual tokenizer to translate the non-linguistic image into a sequence of discrete tokens like a foreign language. The resulting visual tokens encompass high-level semantics worthy of a word and also support dynamic sequence length varying from the image. This unification empowers LaVIT to serve as an impressive generalist interface to understand and generate multi-modal content simultaneously.
  arXiv  Detail & Related papers  (2023-09-09T03:01:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.