Point What You Mean: Visually Grounded Instruction Policy

• URL: http://arxiv.org/abs/2512.18933v1
• Date: Mon, 22 Dec 2025 00:44:19 GMT
• Title: Point What You Mean: Visually Grounded Instruction Policy
• Abstract summary: Point-VLA is a plug-and-play policy that augments language instructions with explicit visual cues to resolve referential ambiguity.<n>We evaluate Point-VLA on diverse real-world referring tasks and observe consistently stronger performance than text-only instruction VLAs.
• Score: 42.52502990975079
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-Language-Action (VLA) models align vision and language with embodied control, but their object referring ability remains limited when relying solely on text prompt, especially in cluttered or out-of-distribution (OOD) scenes. In this study, we introduce the Point-VLA, a plug-and-play policy that augments language instructions with explicit visual cues (e.g., bounding boxes) to resolve referential ambiguity and enable precise object-level grounding. To efficiently scale visually grounded datasets, we further develop an automatic data annotation pipeline requiring minimal human effort. We evaluate Point-VLA on diverse real-world referring tasks and observe consistently stronger performance than text-only instruction VLAs, particularly in cluttered or unseen-object scenarios, with robust generalization. These results demonstrate that Point-VLA effectively resolves object referring ambiguity through pixel-level visual grounding, achieving more generalizable embodied control.

Related papers

• GuideGround: VLM-guided Semantic Understanding and Viewpoint-aware Reasoning for 3D Visual Grounding [11.915446160398625]
  3D visual grounding aims to localize the target object in a 3D scene from a natural language query.<n>We propose GuideGround, a VLM-guided framework that complements rather than replaces task-specific grounding models.
  arXiv  Detail & Related papers  (2026-08-01T08:28:41Z)
• Dynamic Parsing and Updating Natural Language Specification using VLMs for Robust Vision-Language Tracking [61.91472604294714]
  We propose a novel language dependency parsing mechanism to distill tracking principal components.<n>We exploit the powerful cross-modal understanding capability of the pre-trained vision-language model Qwen-VL.<n>Our method achieves consistent and superior tracking performance on multiple large-scale vision-language tracking benchmarks.
  arXiv  Detail & Related papers  (2026-06-28T12:12:18Z)
• AffordVLA: Injecting Affordance Representations into Vision-Language-Action Models via Implicit Feature Alignment [8.122021272122668]
  AffordVLA is an affordance-enhanced Vision-Language-Action (VLA) framework that internalizes manipulation-centric affordance perception into VLA visual representations.<n>We show that AffordVLA effectively reshapes VLA visual representations while preserving inference efficiency, leading to improved manipulation success rates and training efficiency.
  arXiv  Detail & Related papers  (2026-05-17T16:02:05Z)
• VLGOR: Visual-Language Knowledge Guided Offline Reinforcement Learning for Generalizable Agents [14.848584432075285]
  Large Language Models (LLMs) and Reinforcement Learning (RL) enable agents to interpret language instructions more effectively for task execution.<n>We propose Visual-Language Knowledge-Guided Offline Reinforcement Learning (VLGOR), a framework that integrates visual and language knowledge to generate imaginary rollouts.<n> Experiments on robotic manipulation benchmarks demonstrate that VLGOR significantly improves performance on unseen tasks requiring novel optimal policies, achieving a success rate over 24% higher than the baseline methods.
  arXiv  Detail & Related papers  (2026-03-24T07:40:30Z)
• VP-VLA: Visual Prompting as an Interface for Vision-Language-Action Models [80.1055544841585]
  Vision-Language-Action models typically map visual observations and linguistic instructions directly to robotic control signals.<n>We propose VP-VLA, a dual-system framework that decouples high-level reasoning and low-level execution via a structured visual prompting interface.<n> Experiments on the Robocasa-GR1-Tabletop benchmark and SimplerEnv simulation demonstrate that VP-VLA improves success rates by 5% and 8.3%.
  arXiv  Detail & Related papers  (2026-03-23T14:08:58Z)
• Youtu-VL: Unleashing Visual Potential via Unified Vision-Language Supervision [79.06371915084833]
  We introduce Youtu-VL, a framework leveraging the Vision-Language Unified Autoregressive Supervision (VLUAS) paradigm.<n>Youtu-VL applies unified autoregressive supervision to both visual details and linguistic content.<n>We extend this paradigm to encompass vision-centric tasks, enabling a standard VLM to perform vision-centric tasks without task-specific additions.
  arXiv  Detail & Related papers  (2026-01-27T17:01:16Z)
• Clutter-Resistant Vision-Language-Action Models through Object-Centric and Geometry Grounding [14.290592746124835]
  We propose OBEYED-VLA, a framework that disentangles perceptual grounding from action reasoning.<n>OBEYED-VLA augments Vision-Language-Action models with a perception module that grounds inputs into task-conditioned, object-centric, and geometry-aware observations.<n>On a real-world UR10e tabletop setup, OBEYED-VLA substantially improves robustness over strong VLA baselines.
  arXiv  Detail & Related papers  (2025-12-27T08:31:25Z)
• Bring My Cup! Personalizing Vision-Language-Action Models with Visual Attentive Prompting [15.173629384871013]
  Visual Attentive Prompting (VAP) is a training-free perceptual adapter that equips frozen Vision-Language-Action models with selective attention.<n>VAP consistently outperforms generic policies and token-learning baselines in both success rate and correct-object manipulation.
  arXiv  Detail & Related papers  (2025-12-23T03:13:39Z)
• A Multimodal Depth-Aware Method For Embodied Reference Understanding [56.30142869506262]
  Embodied Reference Understanding requires identifying a target object in a visual scene based on both language instructions and pointing cues.<n>We propose a novel ERU framework that jointly leverages data augmentation, depth-map modality, and a depth-aware decision module.
  arXiv  Detail & Related papers  (2025-10-09T14:32:21Z)
• Hierarchical Instruction-aware Embodied Visual Tracking [35.73851196966425]
  User-Centric Embodied Visual Tracking (UC-EVT) presents a novel challenge for reinforcement learning-based models.<n>We propose textbf Instruction-aware Embodied Visual Tracking (HIEVT) agent, which bridges instruction comprehension and action generation using textitspatial goals as intermediaries.
  arXiv  Detail & Related papers  (2025-05-27T04:36:26Z)
• Teaching VLMs to Localize Specific Objects from In-context Examples [56.797110842152]
  We find that present-day Vision-Language Models (VLMs) lack a fundamental cognitive ability: learning to localize specific objects in a scene by taking into account the context.<n>This work is the first to explore and benchmark personalized few-shot localization for VLMs.
  arXiv  Detail & Related papers  (2024-11-20T13:34:22Z)
• OLIVE: Object Level In-Context Visual Embeddings [8.168219870640318]
  We propose a novel method to prompt large language models with in-context visual object vectors. This eliminates the necessity of fusing a lengthy array of image patch features and significantly speeds up training. Our experiments reveal that our method achieves competitive referring object classification and captioning performance.
  arXiv  Detail & Related papers  (2024-06-02T21:36:31Z)
• Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMs [38.02017186215372]
  Integration of Large Language Models (LLMs) into visual domain tasks, resulting in visual-LLMs (V-LLMs), has enabled exceptional performance in vision-language tasks. However, existing V-LLMs demonstrate weak spatial reasoning and localization awareness. We explore how image-space coordinate based instruction fine-tuning objectives could inject spatial awareness into V-LLMs.
  arXiv  Detail & Related papers  (2024-04-11T03:09:34Z)
• Towards Addressing the Misalignment of Object Proposal Evaluation for Vision-Language Tasks via Semantic Grounding [36.03994217853856]
  The performance of object proposals generated for Vision-Language (VL) tasks is currently evaluated across all available annotations. Our work serves as a study of this phenomenon and explores the effectiveness of semantic grounding to mitigate its effects. We show that our method is consistent and demonstrates greatly improved alignment with annotations selected by image captioning metrics and human annotation.
  arXiv  Detail & Related papers  (2023-09-01T02:19:41Z)
• Position-Aware Contrastive Alignment for Referring Image Segmentation [65.16214741785633]
  We present a position-aware contrastive alignment network (PCAN) to enhance the alignment of multi-modal features. Our PCAN consists of two modules: 1) Position Aware Module (PAM), which provides position information of all objects related to natural language descriptions, and 2) Contrastive Language Understanding Module (CLUM), which enhances multi-modal alignment.
  arXiv  Detail & Related papers  (2022-12-27T09:13:19Z)
• PEVL: Position-enhanced Pre-training and Prompt Tuning for Vision-language Models [127.17675443137064]
  We introduce PEVL, which enhances the pre-training and prompt tuning of vision-language models with explicit object position modeling. PEVL reformulates discretized object positions and language in a unified language modeling framework. We show that PEVL enables state-of-the-art performance on position-sensitive tasks such as referring expression comprehension and phrase grounding.
  arXiv  Detail & Related papers  (2022-05-23T10:17:53Z)
• Unsupervised Vision-Language Parsing: Seamlessly Bridging Visual Scene Graphs with Language Structures via Dependency Relationships [17.930724926012264]
  We introduce a new task that targets on inducing a joint vision-language structure in an unsupervised manner. Our goal is to bridge the visual scene graphs and linguistic dependency trees seamlessly. We propose an automatic alignment procedure to produce coarse structures followed by human refinement to produce high-quality ones.
  arXiv  Detail & Related papers  (2022-03-27T09:51:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.