VP-VLA: Visual Prompting as an Interface for Vision-Language-Action Models

• URL: http://arxiv.org/abs/2603.22003v1
• Date: Mon, 23 Mar 2026 14:08:58 GMT
• Title: VP-VLA: Visual Prompting as an Interface for Vision-Language-Action Models
• Abstract summary: Vision-Language-Action models typically map visual observations and linguistic instructions directly to robotic control signals.<n>We propose VP-VLA, a dual-system framework that decouples high-level reasoning and low-level execution via a structured visual prompting interface.<n> Experiments on the Robocasa-GR1-Tabletop benchmark and SimplerEnv simulation demonstrate that VP-VLA improves success rates by 5% and 8.3%.
• Score: 80.1055544841585
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-Language-Action (VLA) models typically map visual observations and linguistic instructions directly to robotic control signals. This "black-box" mapping forces a single forward pass to simultaneously handle instruction interpretation, spatial grounding, and low-level control, often leading to poor spatial precision and limited robustness in out-of-distribution scenarios. To address these limitations, we propose VP-VLA, a dual-system framework that decouples high-level reasoning and low-level execution via a structured visual prompting interface. Specifically, a "System 2 Planner" decomposes complex instructions into sub-tasks and identifies relevant target objects and goal locations. These spatial anchors are then overlaid directly onto visual observations as structured visual prompts, such as crosshairs and bounding boxes. Guided by these prompts and enhanced by a novel auxiliary visual grounding objective during training, a "System 1 Controller" reliably generates precise low-level execution motions. Experiments on the Robocasa-GR1-Tabletop benchmark and SimplerEnv simulation demonstrate that VP-VLA improves success rates by 5% and 8.3%, surpassing competitive baselines including QwenOFT and GR00T-N1.6.

Related papers

• VisualThink-VLA: Visual Intermediate Reasoning for Effective and Low-Latency Vision-Language-Action Policies [58.65913948991329]
  We present VISUALTHINK-VLA, a visual intermediate-reasoning framework for accurate, low-latency VLA policies.<n>Our bootstrapping philosophy is to guide action with effective visual thinking.<n>It bootstraps action prediction through a compact visual-evidence interface that preserves spatial precision while avoiding decoding overhead.
  arXiv  Detail & Related papers  (2026-05-28T14:36:53Z)
• HiVLA: A Visual-Grounded-Centric Hierarchical Embodied Manipulation System [48.02804864175127]
  HiVLA is a visual-grounded-centric hierarchical framework that explicitly decouples high-level semantic planning from low-level motor control.<n>Experiments in simulation and the real world demonstrate that HiVLA significantly outperforms state-of-the-art end-to-end baselines.
  arXiv  Detail & Related papers  (2026-04-15T17:50:07Z)
• DualCoT-VLA: Visual-Linguistic Chain of Thought via Parallel Reasoning for Vision-Language-Action Models [50.07453075750711]
  Vision-Language-Action (VLA) models map visual observations and language instructions directly to robotic actions.<n>Recent efforts have incorporated Chain-of-Thought (CoT) reasoning to endow VLA models with a thinking before acting'' capability.<n>We propose DualCoT-VLA, a visual-linguistic CoT method for VLA models with a parallel reasoning mechanism.
  arXiv  Detail & Related papers  (2026-03-23T17:59:25Z)
• AgentVLN: Towards Agentic Vision-and-Language Navigation [78.739525400071]
  Vision-and-Language Navigation (VLN) requires an embodied agent to ground complex natural-language instructions into long-horizon navigation in unseen environments.<n>In this paper, we propose AgentVLN, a novel and efficient embodied navigation framework that can be deployed on edge computing platforms.
  arXiv  Detail & Related papers  (2026-03-18T12:43:47Z)
• STVG-R1: Incentivizing Instance-Level Reasoning and Grounding in Videos via Reinforcement Learning [65.36458157092207]
  In vision-language models (VLMs), misalignment between textual descriptions and visual coordinates often induces hallucinations.<n>We propose a novel visual prompting paradigm that avoids the difficult problem of aligning coordinates across modalities.<n>We introduce STVG-R1, the first reinforcement learning framework for STVG, which employs a task-driven reward to jointly optimize temporal accuracy, spatial consistency, and structural format regularization.
  arXiv  Detail & Related papers  (2026-02-12T08:53:32Z)
• Point What You Mean: Visually Grounded Instruction Policy [42.52502990975079]
  Point-VLA is a plug-and-play policy that augments language instructions with explicit visual cues to resolve referential ambiguity.<n>We evaluate Point-VLA on diverse real-world referring tasks and observe consistently stronger performance than text-only instruction VLAs.
  arXiv  Detail & Related papers  (2025-12-22T00:44:19Z)
• CogVLA: Cognition-Aligned Vision-Language-Action Model via Instruction-Driven Routing & Sparsification [48.81250395291505]
  Recent Vision-Language-Action models require extensive post-training, resulting in high computational overhead.<n>We propose CogVLA, a framework that leverages instruction-driven routing and sparsification to improve both efficiency and performance.<n>CogVLA achieves state-of-the-art performance with success rates of 97.4% and 70.0%, respectively, while reducing training costs by 2.5-fold and decreasing inference latency by 2.8-fold compared to OpenVLA.
  arXiv  Detail & Related papers  (2025-08-28T17:50:58Z)
• Hierarchical Instruction-aware Embodied Visual Tracking [35.73851196966425]
  User-Centric Embodied Visual Tracking (UC-EVT) presents a novel challenge for reinforcement learning-based models.<n>We propose textbf Instruction-aware Embodied Visual Tracking (HIEVT) agent, which bridges instruction comprehension and action generation using textitspatial goals as intermediaries.
  arXiv  Detail & Related papers  (2025-05-27T04:36:26Z)
• Towards Unified Token Learning for Vision-Language Tracking [65.96561538356315]
  We present a vision-language (VL) tracking pipeline, termed textbfMMTrack, which casts VL tracking as a token generation task. Our proposed framework serializes language description and bounding box into a sequence of discrete tokens. In this new design paradigm, all token queries are required to perceive the desired target and directly predict spatial coordinates of the target.
  arXiv  Detail & Related papers  (2023-08-27T13:17:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.