Steerable Vision-Language-Action Policies for Embodied Reasoning and Hierarchical Control

• URL: http://arxiv.org/abs/2602.13193v2
• Date: Mon, 02 Mar 2026 23:14:20 GMT
• Title: Steerable Vision-Language-Action Policies for Embodied Reasoning and Hierarchical Control
• Abstract summary: We introduce Steerable Policies: VLAs trained on rich synthetic commands at various levels of abstraction, like subtasks, motions, and grounded pixel coordinates.<n>We demonstrate this benefit by controlling our Steerable Policies with both a learned high-level embodied reasoner and an off-the-shelf VLM prompted to reason over command abstractions via in-context learning.
• Score: 46.169163284648384
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Pretrained vision-language models (VLMs) can make semantic and visual inferences across diverse settings, providing valuable common-sense priors for robotic control. However, effectively grounding this knowledge in robot behaviors remains an open challenge. Prior methods often employ a hierarchical approach where VLMs reason over high-level commands to be executed by separate low-level policies, e.g., vision-language-action models (VLAs). The interface between VLMs and VLAs is usually natural language task instructions, which fundamentally limits how much VLM reasoning can steer low-level behavior. We thus introduce Steerable Policies: VLAs trained on rich synthetic commands at various levels of abstraction, like subtasks, motions, and grounded pixel coordinates. By improving low-level controllability, Steerable Policies can unlock pretrained knowledge in VLMs, enabling improved task generalization. We demonstrate this benefit by controlling our Steerable Policies with both a learned high-level embodied reasoner and an off-the-shelf VLM prompted to reason over command abstractions via in-context learning. Across extensive real-world manipulation experiments, these two novel methods outperform prior embodied reasoning VLAs and VLM-based hierarchical baselines, including on challenging generalization and long-horizon tasks. Website: steerable-policies.github.io

Related papers

• Two Bridges, One Pathway: From VLMs to Generalizable VLAs with Embodied Trajectory-Coupled Data [93.7685703383343]
  Vision-language models (VLMs) are powerful general-purpose reasoners, yet converting them into robot control policies is surprisingly difficult.<n>We argue that this gap can be bridged gradually with the right intermediate data.<n>We show that mixing task-relevant out-of-distribution ETC data with a small amount of action data enables the model to generalize to novel visual-language conditions.
  arXiv  Detail & Related papers  (2026-06-07T08:57:51Z)
• QuoVLA: Quotient Space for Vision-Language-Action Models [51.02329790939691]
  Vision-Language-Action (VLA) models commonly adapt pretrained VLM latents to robot control by mapping visual observations and language instructions to continuous actions.<n>Our textitQuotient Theory for VLA shows that pretrained VLM latents are not action-insufficient but action-sufficient.<n>We propose QuoVLA, a quotient-space framework for VLA that compresses pretrained VLM latents into action-sufficient representations.
  arXiv  Detail & Related papers  (2026-05-24T06:28:53Z)
• SteerVLA: Steering Vision-Language-Action Models in Long-Tail Driving Scenarios [104.10555123175055]
  A fundamental challenge in autonomous driving is the integration of high-level, semantic reasoning for long-tail events with low-level, reactive control for robust driving.<n>We propose SteerVLA, which produces fine-grained language instructions that steer a vision-language-action driving policy.<n>We evaluate SteerVLA on a challenging closed-loop benchmark, where it outperforms state-of-the-art methods by 4.77 points in overall driving score and by 8.04 points on a long-tail subset.
  arXiv  Detail & Related papers  (2026-02-09T09:54:02Z)
• How Do VLAs Effectively Inherit from VLMs? [28.72002932514493]
  Vision-language-action (VLA) models hold the promise to attain generalizable embodied control.<n>We introduce a diagnostic benchmark, GrinningFace, an emoji tabletop manipulation task.<n>We investigate the effects of parameter-efficient fine-tuning, VLM freezing, co-training, predicting discretized actions, and predicting latent actions.
  arXiv  Detail & Related papers  (2025-11-10T01:58:02Z)
• From Code to Action: Hierarchical Learning of Diffusion-VLM Policies [8.0703783175731]
  Imitation learning for robotic manipulation often suffers from limited generalization and data scarcity.<n>In this work, we introduce a hierarchical framework that leverages code-generating vision-language models (VLMs)<n>We find that this design enables interpretable policy decomposition, improves generalization when compared to flat policies and enables separate evaluation of high-level planning and low-level control.
  arXiv  Detail & Related papers  (2025-09-29T15:22:18Z)
• From Intention to Execution: Probing the Generalization Boundaries of Vision-Language-Action Models [5.660635614478238]
  Vision-Language-Action (VLA) models promise to produce versatile, "generalist" robot policies.<n>Traditional imitation learning benchmarks are unsuitable due to the lack of language instructions.<n>We introduce a unified suite of 50 simulation-based tasks across 10 subcategories spanning language instruction, vision, and objects.
  arXiv  Detail & Related papers  (2025-06-11T16:52:18Z)
• From Foresight to Forethought: VLM-In-the-Loop Policy Steering via Latent Alignment [11.799979691988902]
  FOREWARN is a novel framework to unlock the potential of Vision Language Models for runtime policy steering.<n>For foresight, we leverage a latent world model to imagine future latent states given diverse low-level action plans.<n>For forethought, we align the VLM with these predicted latent states to reason about the consequences of actions in its native representation.
  arXiv  Detail & Related papers  (2025-02-03T21:11:02Z)
• Robotic Control via Embodied Chain-of-Thought Reasoning [86.6680905262442]
  Key limitation of learned robot control policies is their inability to generalize outside their training data.<n>Recent works on vision-language-action models (VLAs) have shown that the use of large, internet pre-trained vision-language models can substantially improve their robustness and generalization ability.<n>We introduce Embodied Chain-of-Thought Reasoning (ECoT) for VLAs, in which we train VLAs to perform multiple steps of reasoning about plans, sub-tasks, motions, and visually grounded features before predicting the robot action.
  arXiv  Detail & Related papers  (2024-07-11T17:31:01Z)
• From LLMs to Actions: Latent Codes as Bridges in Hierarchical Robot Control [58.72492647570062]
  We introduce our method -- Learnable Latent Codes as Bridges (LCB) -- as an alternate architecture to overcome limitations.<n>We find that methodoutperforms baselines that leverage pure language as the interface layer on tasks that require reasoning and multi-step behaviors.
  arXiv  Detail & Related papers  (2024-05-08T04:14:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.