APIVOT: Adaptive Planning with Interleaved Vision-Language Thoughts

• URL: http://arxiv.org/abs/2607.08024v1
• Date: Thu, 09 Jul 2026 01:02:35 GMT
• Title: APIVOT: Adaptive Planning with Interleaved Vision-Language Thoughts
• Abstract summary: Long-horizon robot planning requires jointly reasoning over semantic task structure and geometric feasibility.<n>We propose APIVOT, a VLM-based planner that adaptively interleaves language and visual thoughts for long-horizon planning.
• Score: 31.4625895558506
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Long-horizon robot planning requires jointly reasoning over semantic task structure and geometric feasibility. To successfully execute a task, a robot must decompose goals, select task-relevant objects, and sequence actions, while ensuring that plans satisfy spatial constraints such as limited free space and object collisions. In this work, we propose APIVOT, a VLM-based planner that adaptively interleaves language and visual thoughts for long-horizon planning. APIVOT learns to leverage language for semantic reasoning, while using visual thoughts as imagined future states for internal verification of geometric feasibility. On long-horizon kitchen tasks, APIVOT outperforms general-purpose VLMs and prior planning frameworks, achieving the largest gains in spatially constrained settings. We find that APIVOT learns meaningful modality selection behavior, demonstrating that adaptive interleaving of vision-language thoughts improves both planning success and reasoning efficiency.

Related papers

• Evidence-Gated Task and Motion Planning with Vision-Language Models [8.674512100396601]
  Under partial observability, the availability of goal-relevant objects may be uncertain.<n>We propose Evidence Acquisition and Feasibility Gating (EAFG)<n>EAFG acquires visual evidence through VLM-generated exploratory subgoals and TAMP-based execution.
  arXiv  Detail & Related papers  (2026-08-20T14:17:52Z)
• WorldScape Policy 2.0: Empowering Steerable World Action Modeling with Reasoning-Augmented Memory [50.19449886428936]
  We introduce WorldScape Policy 2.0, a controllable WAM with reasoning-augmented long short-term memory.<n>The retrieved history augments perception and autoregressively generated planning tokens, yielding an implicit subgoal condition for autonomous planning.<n>Experiments in both simulation and real-world platforms demonstrate superior capabilities in long-horizon autonomous planning, fine-grained instruction following, and in-context adaptation.
  arXiv  Detail & Related papers  (2026-07-21T08:25:37Z)
• Any House Any Task: Scalable Long-Horizon Planning for Abstract Human Tasks [46.59814877000871]
  We propose Any House Any Task (AHAT), a household task planner optimized for long-horizon planning in large environments given ambiguous human instructions.<n>AHAT utilizes an LLM trained to map task instructions and textual scene graphs into grounded subgoals defined in the Planning Domain Definition Language (PDDL)<n>To enhance the model's ability to decompose complex and ambiguous intentions, we introduce TGPO, a novel reinforcement learning algorithm.
  arXiv  Detail & Related papers  (2026-02-12T18:28:28Z)
• Embodied Long Horizon Manipulation with Closed-loop Code Generation and Incremental Few-shot Adaptation [12.077740860502878]
  Embodied long-horizon manipulation requires robotic systems to process multimodal inputs-such as vision and natural language-and translate them into executable actions.<n>Recent methods have explored using large language models (LLMs) as high-level planners that decompose tasks into subtasks using natural language and guide pretrained low-level controllers.<n>Our framework achieves state-of-the-art performance on 30+ diverse seen and unseen long-horizon tasks across LoHoRavens, CALVIN, Franka Kitchen, and cluttered real-world settings.
  arXiv  Detail & Related papers  (2025-03-27T20:32:58Z)
• VSP: Assessing the dual challenges of perception and reasoning in spatial planning tasks for VLMs [102.36953558562436]
  Vision language models (VLMs) are an exciting emerging class of language models (LMs) One understudied capability inVLMs is visual spatial planning. Our study introduces a benchmark that evaluates the spatial planning capability in these models in general.
  arXiv  Detail & Related papers  (2024-07-02T00:24:01Z)
• DKPROMPT: Domain Knowledge Prompting Vision-Language Models for Open-World Planning [9.31108717722043]
  Vision-language models (VLMs) have been applied to robot task planning problems. DKPROMPT automates VLM prompting using domain knowledge in PDDL for classical planning in open worlds.
  arXiv  Detail & Related papers  (2024-06-25T15:49:47Z)
• Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Tasks [50.27313829438866]
  Plan-Seq-Learn (PSL) is a modular approach that uses motion planning to bridge the gap between abstract language and learned low-level control. PSL achieves success rates of over 85%, out-performing language-based, classical, and end-to-end approaches.
  arXiv  Detail & Related papers  (2024-05-02T17:59:31Z)
• Using Left and Right Brains Together: Towards Vision and Language Planning [95.47128850991815]
  We introduce a novel vision-language planning framework to perform concurrent visual and language planning for tasks with inputs of any form. We evaluate the effectiveness of our framework across vision-language tasks, vision-only tasks, and language-only tasks.
  arXiv  Detail & Related papers  (2024-02-16T09:46:20Z)
• Unified Task and Motion Planning using Object-centric Abstractions of Motion Constraints [56.283944756315066]
  We propose an alternative TAMP approach that unifies task and motion planning into a single search. Our approach is based on an object-centric abstraction of motion constraints that permits leveraging the computational efficiency of off-the-shelf AI search to yield physically feasible plans.
  arXiv  Detail & Related papers  (2023-12-29T14:00:20Z)
• Learning adaptive planning representations with natural language guidance [90.24449752926866]
  This paper describes Ada, a framework for automatically constructing task-specific planning representations. Ada interactively learns a library of planner-compatible high-level action abstractions and low-level controllers adapted to a particular domain of planning tasks.
  arXiv  Detail & Related papers  (2023-12-13T23:35:31Z)
• Planning as In-Painting: A Diffusion-Based Embodied Task Planning Framework for Environments under Uncertainty [56.30846158280031]
  Task planning for embodied AI has been one of the most challenging problems. We propose a task-agnostic method named 'planning as in-painting' The proposed framework achieves promising performances in various embodied AI tasks.
  arXiv  Detail & Related papers  (2023-12-02T10:07:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.