Grounding Hierarchical Vision-Language-Action Models Through Explicit Language-Action Alignment

• URL: http://arxiv.org/abs/2604.05614v1
• Date: Tue, 07 Apr 2026 09:03:12 GMT
• Title: Grounding Hierarchical Vision-Language-Action Models Through Explicit Language-Action Alignment
• Abstract summary: To be transparent, a robot's natural language communication must be consistent with its actions.<n>Existing hierarchical Vision-Language-Action (VLA) models can generate language through chain-of-thought and low-level actions.<n>We propose a novel training framework that grounds hierarchical VLA sub-task descriptions with respect to the visual observation and action space.
• Score: 4.001668956134368
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Achieving robot transparency is a critical step toward effective human-robot collaboration. To be transparent, a robot's natural language communication must be consistent with its actions and explicitly grounded in the task and environment. Existing hierarchical Vision-Language-Action (VLA) models can generate language (e.g., through chain-of-thought) and low-level actions. However, current work does not consider explicit alignment between these modalities during training. To address this crucial gap, we propose a novel training framework that explicitly grounds hierarchical VLA sub-task descriptions with respect to the visual observation and action space. Our framework uses a contrastive model to assess the alignment between generated language and corresponding action trajectories. This contrastive model enables direct ranking of different language-trajectory pairs based on their alignment, allowing us to refine the grounding of our hierarchical VLA through offline preference learning. We apply our framework to the LanguageTable dataset, a benchmark dataset of human language-annotated trajectories, and provide critical insights into multimodal grounding representations, all while establishing a strong baseline that achieves performance comparable to fully supervised fine-tuning and minimizing the need for costly data annotations.

Related papers

• In-Context VLA: Endowing Vision-Language-Action Models with Language via In-Context Post-Training and Agentic Tool Use [18.507118969955666]
  Vision-Language-Action models imitate expert action chunks conditioned on a static image and a fixed instruction.<n>We introduce textbfourmethod, a framework that endows a VLA with language competence through in-context post-training.<n>Our method consistently achieves SOTA results in both performance and efficiency when compared with CoT-based approaches.
  arXiv  Detail & Related papers  (2026-08-06T08:21:02Z)
• Task Robustness via Re-Labelling Vision-Action Robot Data [15.985610886484226]
  This paper introduces Task Robustness via Re-Labelling Vision-Action Robot Data (TREAD), a scalable framework to augment existing robotics datasets.<n>Our approach leverages a pretrained VLM through three stages: generating semantic sub-tasks from original instruction labels and initial scenes, segmenting demonstration videos conditioned on these sub-tasks, and producing diverse instructions that incorporate object properties.<n>Our results show that TREAD enhances both planning generalization through trajectory decomposition and language-conditioned policy generalization through increased linguistic diversity.
  arXiv  Detail & Related papers  (2026-06-09T14:28:22Z)
• AffordanceVLA: A Vision-Language-Action Model Empowering Action Generation through Affordance-Aware Understanding [55.535374902204296]
  AffordanceVLA is a unified framework that introduces structured affordance forecasting as a task-oriented intermediate representation.<n>We show that AffordanceVLA achieves strong performance across diverse manipulation scenarios.
  arXiv  Detail & Related papers  (2026-06-04T13:28:51Z)
• VLGOR: Visual-Language Knowledge Guided Offline Reinforcement Learning for Generalizable Agents [14.848584432075285]
  Large Language Models (LLMs) and Reinforcement Learning (RL) enable agents to interpret language instructions more effectively for task execution.<n>We propose Visual-Language Knowledge-Guided Offline Reinforcement Learning (VLGOR), a framework that integrates visual and language knowledge to generate imaginary rollouts.<n> Experiments on robotic manipulation benchmarks demonstrate that VLGOR significantly improves performance on unseen tasks requiring novel optimal policies, achieving a success rate over 24% higher than the baseline methods.
  arXiv  Detail & Related papers  (2026-03-24T07:40:30Z)
• Language-Grounded Decoupled Action Representation for Robotic Manipulation [78.42228162226839]
  We propose the Language-Grounded Decoupled Action Representation (LaDA) framework to connect perception and control.<n>LaDA introduces a fine-grained intermediate layer of three interpretable action primitives--translation, rotation, and gripper control--providing explicit semantic structure for low-level actions.<n>It further employs a semantic-guided soft-label contrastive learning objective to align similar action primitives across tasks, enhancing generalization and motion consistency.
  arXiv  Detail & Related papers  (2026-03-13T13:08:26Z)
• Alignment among Language, Vision and Action Representations [0.0]
  We show that linguistic, visual, and action representations converge toward partially shared semantic structures.<n>These findings indicate that linguistic, visual, and action representations converge toward partially shared semantic structures.
  arXiv  Detail & Related papers  (2026-01-30T13:12:07Z)
• Executable Analytic Concepts as the Missing Link Between VLM Insight and Precise Manipulation [70.8381970762877]
  Vision-Language Models (VLMs) have demonstrated remarkable capabilities in semantic reasoning and task planning.<n>We introduce GRACE, a novel framework that grounds VLM-based reasoning through executable analytic concepts.<n>G GRACE provides a unified and interpretable interface between high-level instruction understanding and low-level robot control.
  arXiv  Detail & Related papers  (2025-10-09T09:08:33Z)
• Do What? Teaching Vision-Language-Action Models to Reject the Impossible [53.40183895299108]
  Vision-Language-Action (VLA) models have demonstrated strong performance on a range of robotic tasks.<n>We propose Instruct-Verify-and-Act (IVA), a framework that detects when an instruction cannot be executed due to a false premise.<n>Our experiments show that IVA improves false premise detection accuracy by 97.56% over baselines.
  arXiv  Detail & Related papers  (2025-08-22T10:54:33Z)
• Temporal and Semantic Evaluation Metrics for Foundation Models in Post-Hoc Analysis of Robotic Sub-tasks [1.8124328823188356]
  We present an automated framework to decompose trajectory data into temporally bounded and natural language-based descriptive sub-tasks.<n>Our framework provides both time-based and language-based descriptions for lower-level sub-tasks that comprise full trajectories.<n>The metrics measure the temporal alignment and semantic fidelity of language descriptions between two sub-task decompositions.
  arXiv  Detail & Related papers  (2024-03-25T22:39:20Z)
• Integrating Self-supervised Speech Model with Pseudo Word-level Targets from Visually-grounded Speech Model [57.78191634042409]
  We propose Pseudo-Word HuBERT (PW-HuBERT), a framework that integrates pseudo word-level targets into the training process. Our experimental results on four spoken language understanding (SLU) benchmarks suggest the superiority of our model in capturing semantic information.
  arXiv  Detail & Related papers  (2024-02-08T16:55:21Z)
• Context-Aware Language Modeling for Goal-Oriented Dialogue Systems [84.65707332816353]
  We formulate goal-oriented dialogue as a partially observed Markov decision process. We derive a simple and effective method to finetune language models in a goal-aware way. We evaluate our method on a practical flight-booking task using AirDialogue.
  arXiv  Detail & Related papers  (2022-04-18T17:23:11Z)
• ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning [97.10875695679499]
  We propose a novel contrastive learning framework named ERICA in pre-training phase to obtain a deeper understanding of the entities and their relations in text. Experimental results demonstrate that our proposed ERICA framework achieves consistent improvements on several document-level language understanding tasks.
  arXiv  Detail & Related papers  (2020-12-30T03:35:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.