GIVE: Grounding Human Gestures in Vision-Language-Action Models

• URL: http://arxiv.org/abs/2606.13435v1
• Date: Thu, 11 Jun 2026 14:59:38 GMT
• Title: GIVE: Grounding Human Gestures in Vision-Language-Action Models
• Abstract summary: GIVE (Gesture Intent via Visual-Semantic Enhancement) is an effective approach to enhance pre-trained Vision-Language-Action (VLA) models with human gesture understanding.<n>In real-world HRI experiments, GIVE substantially outperforms the baseline, improving target object recognition accuracy by 40% and overall task success rate by 80%.
• Score: 38.010995423052975
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Human communication is inherently multimodal, where language is often accompanied by non-verbal cues such as gestures to convey intentions. However, current Vision-Language-Action (VLA) models treat robotic manipulation as a pure text-driven task, overlooking the important role of gestures in Human-Robot Interaction (HRI). This often leads to inaccurate intent grounding and unreliable manipulation when language instructions are ambiguous or underspecified. To address this challenge, we propose GIVE (Gesture Intent via Visual-Semantic Enhancement), an effective approach that enhances pre-trained VLA models with human gesture understanding without architectural modifications. Specifically, GIVE incorporates gesture information through two complementary pathways: a visual pathway that overlays hand skeletons and fingertip rays onto robot observations for explicit object grounding, and a semantic pathway that generates high-level descriptions of human gestures and task instructions for robust intent grounding. By jointly leveraging visual and semantic guidance, GIVE enables VLA policies to better associate gestures with manipulation behaviors and adapt to dynamic interaction intents. In real-world HRI experiments, GIVE substantially outperforms the baseline, improving target object recognition accuracy by 40% and overall task success rate by 80%, while demonstrating strong robustness and generalization to unseen spatial layouts and diverse participants.

Related papers

• HINT: Human-Intent Inception for Long-Horizon Robot Manipulation [11.55318573740128]
  We present HINT (Human-INTent INcepTion), an agentic framework inspired by the human manipulation principles.<n>We show that HINT substantially improves intent understanding, task progress, and end-to-end success across two foundation policies.
  arXiv  Detail & Related papers  (2026-09-02T14:26:30Z)
• Gaze2Act: Gaze-Conditioned Vision-Language-Action Policies for Interactive Robot Manipulation [24.026684861369386]
  Vision-Language-Action (VLA) models have recently shown strong potential for robot learning by following language instructions.<n>It is difficult to describe which exact object to interact with among similar candidates, where to act on the object, or how the target may change during execution.<n>We propose Gaze2Act, a novel VLA framework that leverages human gaze as a dynamic and intuitive intent signal for complex interactive manipulation.
  arXiv  Detail & Related papers  (2026-05-28T17:37:16Z)
• GesVLA: Gesture-Aware Vision-Language-Action Model Embedded Representations [19.347332945780273]
  Vision-Language-Action (VLA) models have shown strong potential for general-purpose robot manipulation by unifying perception and action.<n>We introduce gesture as a parallel instruction modality and propose a Gesture-aware Vision-Language-Action model (GesVLA)<n> Experimental results show that incorporating gesture consistently improves target grounding accuracy and human-robot interaction efficiency.
  arXiv  Detail & Related papers  (2026-05-21T17:57:44Z)
• IntentVLM: Open-Vocabulary Intention Recognition through Forward-Inverse Modeling with Video-Language Models [5.381126267099941]
  This paper presents IntentVLM, a two-stage video-language framework for open-vocabulary human intention recognition.<n>IntrepidVLM is inspired by forward-inverse modeling in cognitive science by decomposing intention understanding into goal candidate generation.<n>It achieves state-of-the-art results with up to 80% accuracy, notably surpassing the baseline performance by 30% and matches human performance.
  arXiv  Detail & Related papers  (2026-04-27T03:34:33Z)
• Grounding Hierarchical Vision-Language-Action Models Through Explicit Language-Action Alignment [4.001668956134368]
  To be transparent, a robot's natural language communication must be consistent with its actions.<n>Existing hierarchical Vision-Language-Action (VLA) models can generate language through chain-of-thought and low-level actions.<n>We propose a novel training framework that grounds hierarchical VLA sub-task descriptions with respect to the visual observation and action space.
  arXiv  Detail & Related papers  (2026-04-07T09:03:12Z)
• Language-Grounded Decoupled Action Representation for Robotic Manipulation [78.42228162226839]
  We propose the Language-Grounded Decoupled Action Representation (LaDA) framework to connect perception and control.<n>LaDA introduces a fine-grained intermediate layer of three interpretable action primitives--translation, rotation, and gripper control--providing explicit semantic structure for low-level actions.<n>It further employs a semantic-guided soft-label contrastive learning objective to align similar action primitives across tasks, enhancing generalization and motion consistency.
  arXiv  Detail & Related papers  (2026-03-13T13:08:26Z)
• Executable Analytic Concepts as the Missing Link Between VLM Insight and Precise Manipulation [70.8381970762877]
  Vision-Language Models (VLMs) have demonstrated remarkable capabilities in semantic reasoning and task planning.<n>We introduce GRACE, a novel framework that grounds VLM-based reasoning through executable analytic concepts.<n>G GRACE provides a unified and interpretable interface between high-level instruction understanding and low-level robot control.
  arXiv  Detail & Related papers  (2025-10-09T09:08:33Z)
• GoViG: Goal-Conditioned Visual Navigation Instruction Generation [69.79110149746506]
  We introduce Goal-Conditioned Visual Navigation Instruction Generation (GoViG), a new task that aims to autonomously generate precise and contextually coherent navigation instructions.<n>GoViG exclusively leverages raw egocentric visual data, substantially improving its adaptability to unseen and unstructured environments.
  arXiv  Detail & Related papers  (2025-08-13T07:05:17Z)
• Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous Sensors via Language Grounding [85.63710017456792]
  FuSe is a novel approach that enables finetuning visuomotor generalist policies on heterogeneous sensor modalities.<n>We show that FuSe enables performing challenging tasks that require reasoning jointly over modalities such as vision, touch, and sound.<n>Experiments in the real world show that FuSeis able to increase success rates by over 20% compared to all considered baselines.
  arXiv  Detail & Related papers  (2025-01-08T18:57:33Z)
• Human-oriented Representation Learning for Robotic Manipulation [64.59499047836637]
  Humans inherently possess generalizable visual representations that empower them to efficiently explore and interact with the environments in manipulation tasks. We formalize this idea through the lens of human-oriented multi-task fine-tuning on top of pre-trained visual encoders. Our Task Fusion Decoder consistently improves the representation of three state-of-the-art visual encoders for downstream manipulation policy-learning.
  arXiv  Detail & Related papers  (2023-10-04T17:59:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.