Evidence-Gated Task and Motion Planning with Vision-Language Models

• URL: http://arxiv.org/abs/2608.20084v1
• Date: Thu, 20 Aug 2026 14:17:52 GMT
• Title: Evidence-Gated Task and Motion Planning with Vision-Language Models
• Abstract summary: Under partial observability, the availability of goal-relevant objects may be uncertain.<n>We propose Evidence Acquisition and Feasibility Gating (EAFG)<n>EAFG acquires visual evidence through VLM-generated exploratory subgoals and TAMP-based execution.
• Score: 8.674512100396601
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Robots executing long-horizon manipulation tasks from natural-language instructions must reason about both semantic task structure and geometric feasibility. However, under partial observability, the availability of goal-relevant objects may be uncertain. In such cases, approaches that combine Vision-Language Models (VLMs) with Task and Motion Planning (TAMP) may generate subgoals that rely on the VLM's prior knowledge without observational support, leading to execution failures or unintended outcomes. We propose Evidence Acquisition and Feasibility Gating (EAFG), a framework that acquires visual evidence through VLM-generated exploratory subgoals and TAMP-based execution. EAFG then applies a feasibility gate to decide whether to proceed with task planning, acquire further evidence, or halt. Our experiments show that, in cooking tasks with ambiguous object use, EAFG improves recipe completion by discovering task-relevant objects before planning. For instructions requiring an absent object, EAFG promotes appropriate halt decisions and reduces repeated attempts to manipulate that object.

Related papers

• Plan Along the Way: Event-Triggered Foundation-Model Planning for TAMP Execution in Partially Observable Manipulation [1.37013665345905]
  ROBUST TAMP is a modular LLM/VLM-guided planning framework for reactive TAMP.<n>System reconstructs the visible scene state and replans using completed-action history and structured replanning event context.<n> Evaluations are performed on six RLBench/CoppeliaSim kitchen and grill variants involving hidden objects, non-target object discovery, articulated-container interaction, and temporal manipulation procedures.
  arXiv  Detail & Related papers  (2026-08-28T08:44:47Z)
• APIVOT: Adaptive Planning with Interleaved Vision-Language Thoughts [31.4625895558506]
  Long-horizon robot planning requires jointly reasoning over semantic task structure and geometric feasibility.<n>We propose APIVOT, a VLM-based planner that adaptively interleaves language and visual thoughts for long-horizon planning.
  arXiv  Detail & Related papers  (2026-07-09T01:02:35Z)
• EV-WM: Event-Verified World Models for Long-Horizon Robotic Manipulation [0.815557531820863]
  Long-horizon manipulation requires progress signals that are relational, predicate-level, and physically grounded.<n>We introduce textbfEV-WM, a predicate-grounded verification framework for world-model planning.
  arXiv  Detail & Related papers  (2026-06-11T08:35:37Z)
• PRTS: A Primitive Reasoning and Tasking System via Contrastive Representations [66.94988600664574]
  We present textbfPRTS (textbfPrimitive textbfReasoning and textbfTasking textbfSystem), a VLA foundation model that reformulates pretraining through Goal-Conditioned Reinforcement Learning.
  arXiv  Detail & Related papers  (2026-04-30T06:14:02Z)
• Effective Task Planning with Missing Objects using Learning-Informed Object Search [12.369687852088859]
  We develop a planning framework centered around novel model-based LIOS actions.<n>High-level planning treats LIOS actions as deterministic and so -- informed by model-based calculations of the expected cost of each -- generates plans that interleave search and execution.<n>In simulated ProcTHOR homes and in the real world, our approach outperforms non-learned and learned baselines on tasks including retrieval and meal prep.
  arXiv  Detail & Related papers  (2026-02-12T00:56:35Z)
• Subgoal Graph-Augmented Planning for LLM-Guided Open-World Reinforcement Learning [0.0]
  Large language models (LLMs) offer strong high-level planning capabilities for reinforcement learning.<n>LLMs produce subgoals that are semantically plausible but infeasible or irrelevant in the target environment.<n>LLMs planning conflates generation with self-verification, resulting in overconfident yet unreliable subgoals.
  arXiv  Detail & Related papers  (2025-11-26T02:49:44Z)
• Learning Affordances at Inference-Time for Vision-Language-Action Models [50.93181349331096]
  In robotics, Vision-Language-Action models (VLAs) offer a promising path towards solving complex control tasks.<n>We introduce Learning from Inference-Time Execution (LITEN), which connects a VLA low-level policy to a high-level VLM that conditions on past experiences.<n>Our approach iterates between a reasoning phase that generates and executes plans for the low-level VLA, and an assessment phase that reflects on the resulting execution.
  arXiv  Detail & Related papers  (2025-10-22T16:43:29Z)
• LLM Agents Beyond Utility: An Open-Ended Perspective [50.809163251551894]
  We augment a pretrained LLM agent with the ability to generate its own tasks, accumulate knowledge, and interact extensively with its environment.<n>It can reliably follow complex multi-step instructions, store and reuse information across runs, and propose and solve its own tasks.<n>It remains sensitive to prompt design, prone to repetitive task generation, and unable to form self-representations.
  arXiv  Detail & Related papers  (2025-10-16T10:46:54Z)
• LangPert: Detecting and Handling Task-level Perturbations for Robust Object Rearrangement [21.236557779562794]
  LangPert is a language-based framework designed to detect and mitigate Task-Level Perturbations (TLP)<n>LangPert integrates a Visual Language Model (VLM) to comprehensively monitor policy's skill execution and environmental TLP.<n>Our experimental results demonstrate that LangPert handles diverse TLP situations more effectively than baseline methods.
  arXiv  Detail & Related papers  (2025-04-14T05:39:15Z)
• Propose, Assess, Search: Harnessing LLMs for Goal-Oriented Planning in Instructional Videos [48.15438373870542]
  VidAssist is an integrated framework designed for zero/few-shot goal-oriented planning in instructional videos. It employs a breadth-first search algorithm for optimal plan generation. Experiments demonstrate that VidAssist offers a unified framework for different goal-oriented planning setups.
  arXiv  Detail & Related papers  (2024-09-30T17:57:28Z)
• VSP: Assessing the dual challenges of perception and reasoning in spatial planning tasks for VLMs [102.36953558562436]
  Vision language models (VLMs) are an exciting emerging class of language models (LMs) One understudied capability inVLMs is visual spatial planning. Our study introduces a benchmark that evaluates the spatial planning capability in these models in general.
  arXiv  Detail & Related papers  (2024-07-02T00:24:01Z)
• Unified Task and Motion Planning using Object-centric Abstractions of Motion Constraints [56.283944756315066]
  We propose an alternative TAMP approach that unifies task and motion planning into a single search. Our approach is based on an object-centric abstraction of motion constraints that permits leveraging the computational efficiency of off-the-shelf AI search to yield physically feasible plans.
  arXiv  Detail & Related papers  (2023-12-29T14:00:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.