Learning Structured Robot Policies from Vision-Language Models via Synthetic Neuro-Symbolic Supervision

• URL: http://arxiv.org/abs/2604.02812v1
• Date: Fri, 03 Apr 2026 07:27:33 GMT
• Title: Learning Structured Robot Policies from Vision-Language Models via Synthetic Neuro-Symbolic Supervision
• Abstract summary: Vision-language models (VLMs) have recently demonstrated strong capabilities in mapping multimodal observations to robot behaviors.<n>Most current approaches rely on end-to-end visuomotor policies that remain opaque and difficult to analyze.<n>This work investigates how foundation models can be specialized to generate structured robot policies grounded in multimodal perception.
• Score: 39.03634103868044
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-language models (VLMs) have recently demonstrated strong capabilities in mapping multimodal observations to robot behaviors. However, most current approaches rely on end-to-end visuomotor policies that remain opaque and difficult to analyze, limiting their use in safety-critical robotic applications. In contrast, classical robotic systems often rely on structured policy representations that provide interpretability, modularity, and reactive execution. This work investigates how foundation models can be specialized to generate structured robot policies grounded in multimodal perception, bridging high-dimensional learning and symbolic control. We propose a neuro-symbolic approach in which a VLM synthesizes executable Behavior Tree policies from visual observations, natural language instructions, and structured system specifications. To enable scalable supervision without manual annotation, we introduce an automated pipeline that generates a synthetic multimodal dataset of domain-randomized scenes paired with instruction-policy examples produced by a foundation model. Real-world experiments on two robotic manipulators show that structured policies learned entirely from synthetic supervision transfer successfully to physical systems. The results indicate that foundation models can be adapted to produce interpretable and structured robot policies, providing an alternative to opaque end-to-end approaches for multimodal robot decision making.

Related papers

• A Few Words Go a Long Way: Language Guided Robot Policy Synthesis [9.088404024682706]
  ARCHITECT is a framework that treats robot policy acquisition as an interactive program synthesis task.<n>We introduce an iterative process where a human supervisor provides natural language corrections to steer the policy.<n>In a benchmark evaluation on a Franka Panda robot, ARCHITECT outperforms state-of-the-art VLA models.
  arXiv  Detail & Related papers  (2026-07-26T18:00:02Z)
• Decoupling the Declarative from the Procedural in Vision-Language-Action Models [65.22639791239932]
  Policy trained to clone a behavior from object-specific demonstrations must generalize beyond that object.<n>We propose w$2$VLA, a new Vision-Language-Action model with restructured information flow.<n>Unlike popular, state-of-the-art VLAs, we show that our modular approach successfully decouples knowledge representations.
  arXiv  Detail & Related papers  (2026-06-19T14:43:58Z)
• Build on Priors: Vision--Language--Guided Neuro-Symbolic Imitation Learning for Data-Efficient Real-World Robot Manipulation [4.118262876469644]
  We present a scalable neuro-symbolic framework that autonomously constructs symbolic planning domains and data-efficient control policies.<n>Our method segments demonstrations into skills and employs a Vision-Language Model (VLM) to classify skills.<n>Known controllers can be leveraged for real-world data augmentation by projecting a single demonstration onto other objects in the scene.
  arXiv  Detail & Related papers  (2026-04-04T15:17:59Z)
• Closed-Loop Verbal Reinforcement Learning for Task-Level Robotic Planning [1.6641413441634143]
  We propose a new Verbal Reinforcement Learning framework for interpretable task-level planning in mobile robotic systems.<n>The framework follows a closed-loop architecture that enables iterative policy improvement through interaction with the physical environment.<n>We validate the proposed framework on a real mobile robot performing a multi-stage manipulation and navigation task under execution uncertainty.
  arXiv  Detail & Related papers  (2026-03-23T16:28:36Z)
• Evaluating Gemini Robotics Policies in a Veo World Simulator [69.23071832313246]
  We introduce a generative evaluation system built upon a frontier video foundation model (Veo)<n>The system is optimized to support robot action conditioning and multi-view consistency.<n>We validate these capabilities through 1600+ real-world evaluations of eight Gemini Robotics policy checkpoints and five tasks for a bimanual manipulator.
  arXiv  Detail & Related papers  (2025-12-11T14:22:14Z)
• Bridging Embodiment Gaps: Deploying Vision-Language-Action Models on Soft Robots [5.993870098970107]
  Vision-Language-Action (VLA) models have been proposed as a language guided generalized control framework for real robots.<n>We present the deployment of a VLA model on a soft continuum manipulator to demonstrate autonomous safe human-robot interaction.
  arXiv  Detail & Related papers  (2025-10-20T10:06:39Z)
• $π_0$: A Vision-Language-Action Flow Model for General Robot Control [77.32743739202543]
  We propose a novel flow matching architecture built on top of a pre-trained vision-language model (VLM) to inherit Internet-scale semantic knowledge. We evaluate our model in terms of its ability to perform tasks in zero shot after pre-training, follow language instructions from people, and its ability to acquire new skills via fine-tuning.
  arXiv  Detail & Related papers  (2024-10-31T17:22:30Z)
• GRAPPA: Generalizing and Adapting Robot Policies via Online Agentic Guidance [15.774237279917594]
  We propose an agentic framework for robot self-guidance and self-improvement.<n>Our framework iteratively grounds a base robot policy to relevant objects in the environment.<n>We demonstrate that our approach can effectively guide manipulation policies to achieve significantly higher success rates.
  arXiv  Detail & Related papers  (2024-10-09T02:00:37Z)
• RoboCodeX: Multimodal Code Generation for Robotic Behavior Synthesis [102.1876259853457]
  We propose a tree-structured multimodal code generation framework for generalized robotic behavior synthesis, termed RoboCodeX. RoboCodeX decomposes high-level human instructions into multiple object-centric manipulation units consisting of physical preferences such as affordance and safety constraints. To further enhance the capability to map conceptual and perceptual understanding into control commands, a specialized multimodal reasoning dataset is collected for pre-training and an iterative self-updating methodology is introduced for supervised fine-tuning.
  arXiv  Detail & Related papers  (2024-02-25T15:31:43Z)
• Transferring Foundation Models for Generalizable Robotic Manipulation [82.12754319808197]
  We propose a novel paradigm that effectively leverages language-reasoning segmentation mask generated by internet-scale foundation models.<n>Our approach can effectively and robustly perceive object pose and enable sample-efficient generalization learning.<n>Demos can be found in our submitted video, and more comprehensive ones can be found in link1 or link2.
  arXiv  Detail & Related papers  (2023-06-09T07:22:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.