SignVLA: Real-Time Sign Language-Guided Robotic Manipulation via Attention LSTM and Vision-Language-Action Models

• URL: http://arxiv.org/abs/2606.20857v1
• Date: Thu, 18 Jun 2026 18:46:12 GMT
• Title: SignVLA: Real-Time Sign Language-Guided Robotic Manipulation via Attention LSTM and Vision-Language-Action Models
• Abstract summary: Vision-Language-Action (VLA) models enable robots to execute manipulation tasks from natural-language instructions grounded in visual observations.<n>We present SignVLA, a real-time sign-language-guided VLA framework for accessible human-robot interaction.
• Score: 1.4175612723267692
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-Language-Action (VLA) models enable robots to execute manipulation tasks from natural-language instructions grounded in visual observations. However, existing VLA interfaces primarily rely on speech or text input, limiting accessibility for deaf, hard-of-hearing, and speech-impaired users. We present SignVLA, a real-time sign-language-guided VLA framework for accessible human-robot interaction. The system introduces a modular sign-to-text interface that converts visual sign gestures into semantic instructions compatible with downstream VLA policies. Given video streams, SignVLA extracts hand landmark features and employs an attention-enhanced Long Short-Term Memory (LSTM) network to capture temporal gesture dynamics for alphabet- and command-level sign recognition. A temporal stabilization module further improves prediction consistency in real-time interaction settings.The generated instruction sequence is then passed to a downstream VLA policy for sign-conditioned robotic manipulation. Experimental results demonstrate stable real-time sign recognition and successful execution of manipulation tasks driven by sign-language inputs. Our findings suggest that lightweight temporal sign recognition can serve as an effective and practical accessibility layer for multimodal embodied intelligence.

Related papers

• SMART: MLLM-guided Temporal Alignment for Unifying Sign Language Recognition and Spotting [1.3287545480967504]
  We propose an MLLM-guided temporal alignment framework for joint sign recognition and spotting.<n> SMART uses motion descriptions as auxiliary semantic cues and performs stable videotext alignment under small-batch training.<n>Experiments on four sign language benchmarks demonstrate the effectiveness of SMART across both recognition and spotting tasks.
  arXiv  Detail & Related papers  (2026-08-26T08:06:10Z)
• ASCII Art Turns LLMs into VLA Controllers [24.00781789697405]
  Vision--Language--Action (VLA) controllers are often built by extending vision--language models (VLMs) with action supervision.<n>We demonstrate that a text-only large language model (LLM) can be adapted into a VLA-style controller when visual observations are rendered into a text input.
  arXiv  Detail & Related papers  (2026-06-19T14:19:59Z)
• Toward Signing Activity Projection in Sign Language Interaction [7.198816748075489]
  This paper presents an initial transfer study of adapting a VAP architecture to dyadic sign language interaction.<n>We derive binary signing activity streams from lexical sign annotations and formulate proxy tasks for turn-taking prediction.<n>The results show that SHIFT/HOLD prediction is promising, especially with hand cues, while SHIFT-prediction remains difficult.
  arXiv  Detail & Related papers  (2026-06-08T12:38:09Z)
• QuoVLA: Quotient Space for Vision-Language-Action Models [51.02329790939691]
  Vision-Language-Action (VLA) models commonly adapt pretrained VLM latents to robot control by mapping visual observations and language instructions to continuous actions.<n>Our textitQuotient Theory for VLA shows that pretrained VLM latents are not action-insufficient but action-sufficient.<n>We propose QuoVLA, a quotient-space framework for VLA that compresses pretrained VLM latents into action-sufficient representations.
  arXiv  Detail & Related papers  (2026-05-24T06:28:53Z)
• SignVLA: A Gloss-Free Vision-Language-Action Framework for Real-Time Sign Language-Guided Robotic Manipulation [1.4175612723267692]
  We present the first sign language-driven Vision-Language-Action (VLA) framework for intuitive human-robot interaction.<n>Unlike conventional approaches that rely on gloss annotations as intermediate supervision, the proposed system adopts a gloss-free paradigm.<n>We focus on a real-time alphabet-level finger-spelling interface that provides a robust and low-latency communication channel for robotic control.
  arXiv  Detail & Related papers  (2026-02-26T01:16:27Z)
• LoLA: Long Horizon Latent Action Learning for General Robot Manipulation [38.209790870296835]
  Long Horizon Latent Action Learning (LoLA) is a framework designed for robot manipulation.<n>We first employ Vision-Language Models to encode rich contextual features from historical sequences and multi-view observations.<n>We then introduce State-Aware Latent Re-representation, which transforms visual inputs and language commands into actionable robot motion space.
  arXiv  Detail & Related papers  (2025-12-23T08:45:24Z)
• Prompting with Sign Parameters for Low-resource Sign Language Instruction Generation [2.6171111742555015]
  Sign Language (SL) enables two-way communication for the deaf and hard-of-hearing community.<n>Many sign languages remain under-resourced in the AI space.<n>We produce step-by-step textual instructions that enable non-SL users to imitate and learn SL gestures.
  arXiv  Detail & Related papers  (2025-08-22T04:11:28Z)
• Signs as Tokens: A Retrieval-Enhanced Multilingual Sign Language Generator [55.94334001112357]
  We introduce a multilingual sign language model, Signs as Tokens (SOKE), which can generate 3D sign avatars autoregressively from text inputs.<n>We propose a retrieval-enhanced SLG approach, which incorporates external sign dictionaries to provide accurate word-level signs.
  arXiv  Detail & Related papers  (2024-11-26T18:28:09Z)
• SHuBERT: Self-Supervised Sign Language Representation Learning via Multi-Stream Cluster Prediction [65.1590372072555]
  SHuBERT (Sign Hidden-Unit BERT) is a self-supervised contextual representation model learned from 1,000 hours of American Sign Language video.<n>SHuBERT adapts masked token prediction objectives to multi-stream visual sign language input, learning to predict multiple targets corresponding to clustered hand, face, and body pose streams.<n>SHuBERT achieves state-of-the-art performance across multiple tasks including sign language translation, isolated sign language recognition, and fingerspelling detection.
  arXiv  Detail & Related papers  (2024-11-25T03:13:08Z)
• EvSign: Sign Language Recognition and Translation with Streaming Events [59.51655336911345]
  Event camera could naturally perceive dynamic hand movements, providing rich manual clues for sign language tasks. We propose efficient transformer-based framework for event-based SLR and SLT tasks. Our method performs favorably against existing state-of-the-art approaches with only 0.34% computational cost.
  arXiv  Detail & Related papers  (2024-07-17T14:16:35Z)
• LLaRA: Supercharging Robot Learning Data for Vision-Language Policy [56.505551117094534]
  We introduce LLaRA: Large Language and Robotics Assistant, a framework that formulates robot action policy as visuo-textual conversations.<n>First, we present an automated pipeline to generate conversation-style instruction tuning data for robots from existing behavior cloning datasets.<n>We show that a VLM finetuned with a limited amount of such datasets can produce meaningful action decisions for robotic control.
  arXiv  Detail & Related papers  (2024-06-28T17:59:12Z)
• SignVTCL: Multi-Modal Continuous Sign Language Recognition Enhanced by Visual-Textual Contrastive Learning [51.800031281177105]
  SignVTCL is a continuous sign language recognition framework enhanced by visual-textual contrastive learning. It integrates multi-modal data (video, keypoints, and optical flow) simultaneously to train a unified visual backbone. It achieves state-of-the-art results compared with previous methods.
  arXiv  Detail & Related papers  (2024-01-22T11:04:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.