Large-Language-Model-Guided State Estimation for Partially Observable Task and Motion Planning

• URL: http://arxiv.org/abs/2603.03704v2
• Date: Fri, 06 Mar 2026 00:16:10 GMT
• Title: Large-Language-Model-Guided State Estimation for Partially Observable Task and Motion Planning
• Abstract summary: We propose incorporating two types of common-sense knowledge: (1) certain objects are more likely to be found in specific locations; and (2) similar objects are likely to be co-located, while dissimilar objects are less likely to be found together.<n>CoCo-TAMP achieves an average reduction of 62.7% in planning and execution time in simulation, and 72.6% in real-world demonstrations.
• Score: 30.00573299632629
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Robot planning in partially observable environments, where not all objects are known or visible, is a challenging problem, as it requires reasoning under uncertainty through partially observable Markov decision processes. During the execution of a computed plan, a robot may unexpectedly observe task-irrelevant objects, which are typically ignored by naive planners. In this work, we propose incorporating two types of common-sense knowledge: (1) certain objects are more likely to be found in specific locations; and (2) similar objects are likely to be co-located, while dissimilar objects are less likely to be found together. Manually engineering such knowledge is complex, so we explore leveraging the powerful common-sense reasoning capabilities of large language models (LLMs). Our planning and execution framework, CoCo-TAMP, introduces a hierarchical state estimation that uses LLM-guided information to shape the belief over task-relevant objects, enabling efficient solutions to long-horizon task and motion planning problems. In experiments, CoCo-TAMP achieves an average reduction of 62.7% in planning and execution time in simulation, and 72.6% in real-world demonstrations, compared to a baseline that does not incorporate either type of common-sense knowledge.

Related papers

• Leveraging Inter-object Affordances for Efficient Planning in Contact-rich Tasks [42.29257983447145]
  Traditional task-and-motion planning (TAMP) approaches focus on defining sequences of actions along with the necessary geometric and kinematic constraints.<n>We propose a method that leverages a TAMP approach, defining object-centric abstractions of execution constraints.<n>Our approach achieves significantly higher planning success rates and improves planning times by one to two orders of magnitude compared to other methods.
  arXiv  Detail & Related papers  (2026-08-26T11:12:46Z)
• Generalizable task-oriented object grasping through LLM-guided ontology and similarity-based planning [16.214119457569534]
  Task-oriented grasping (TOG) is more challenging than simple object grasping because it requires precise identification of object parts and careful selection of grasping areas to ensure effective and robust manipulation.<n>We introduce a novel, geometry-centric strategy for more generalizable TOG that does not rely on semantic features from visual recognition.<n>We validate the high accuracy of our approach in functional part selection, identification, and grasp generation through real-world experiments.
  arXiv  Detail & Related papers  (2026-03-27T13:39:57Z)
• Scale-Plan: Scalable Language-Enabled Task Planning for Heterogeneous Multi-Robot Teams [11.964765256346857]
  Long-horizon task planning for heterogeneous multi-robot systems is essential for deploying collaborative teams in real-world environments.<n>We present Scale-Plan, a scalable LLM-assisted framework that generates compact, task-relevant problem representations from natural language instructions.<n>By filtering irrelevant information prior to planning, Scale-Plan enables efficient decomposition, allocation, and long-horizon plan generation.
  arXiv  Detail & Related papers  (2026-03-09T18:13:18Z)
• Effective Task Planning with Missing Objects using Learning-Informed Object Search [12.369687852088859]
  We develop a planning framework centered around novel model-based LIOS actions.<n>High-level planning treats LIOS actions as deterministic and so -- informed by model-based calculations of the expected cost of each -- generates plans that interleave search and execution.<n>In simulated ProcTHOR homes and in the real world, our approach outperforms non-learned and learned baselines on tasks including retrieval and meal prep.
  arXiv  Detail & Related papers  (2026-02-12T00:56:35Z)
• OmniEVA: Embodied Versatile Planner via Task-Adaptive 3D-Grounded and Embodiment-aware Reasoning [50.45036742963495]
  We introduce OmniEVA, an embodied versatile planner that enables advanced embodied reasoning and task planning.<n>A Task-Adaptive 3D Grounding mechanism enables context-aware 3D grounding for diverse embodied tasks.<n>An Embodiment-Aware Reasoning framework incorporates task goals and embodiment constraints into the reasoning loop, resulting in planning decisions that are both goal-directed and executable.
  arXiv  Detail & Related papers  (2025-09-11T10:32:22Z)
• ThinkGeo: Evaluating Tool-Augmented Agents for Remote Sensing Tasks [64.86209459039313]
  ThinkGeo is an agentic benchmark designed to evaluate tool-augmented agents on remote sensing tasks via structured tool use and multi-step planning.<n>We implement a ReAct-style interaction loop and evaluate both open and closed-source LLMs on 486 structured agentic tasks with 1,773 expert-verified reasoning steps.<n>Our analysis reveals notable disparities in tool accuracy and planning consistency across models.
  arXiv  Detail & Related papers  (2025-05-29T17:59:38Z)
• VSP: Assessing the dual challenges of perception and reasoning in spatial planning tasks for VLMs [102.36953558562436]
  Vision language models (VLMs) are an exciting emerging class of language models (LMs) One understudied capability inVLMs is visual spatial planning. Our study introduces a benchmark that evaluates the spatial planning capability in these models in general.
  arXiv  Detail & Related papers  (2024-07-02T00:24:01Z)
• Embodied Instruction Following in Unknown Environments [64.57388036567461]
  We propose an embodied instruction following (EIF) method for complex tasks in the unknown environment.<n>We build a hierarchical embodied instruction following framework including the high-level task planner and the low-level exploration controller.<n>For the task planner, we generate the feasible step-by-step plans for human goal accomplishment according to the task completion process and the known visual clues.
  arXiv  Detail & Related papers  (2024-06-17T17:55:40Z)
• Details Make a Difference: Object State-Sensitive Neurorobotic Task Planning [15.03025428687218]
  The state of an object reflects its current status or condition and is important for a robot's task planning and manipulation. Recently, pre-trained Large Language Models (LLMs) and Vision-Language Models (VLMs) have shown impressive capabilities in generating plans. We introduce an Object State-Sensitive Agent (OSSA), a task-planning agent empowered by pre-trained neural networks.
  arXiv  Detail & Related papers  (2024-06-14T12:52:42Z)
• Unified Task and Motion Planning using Object-centric Abstractions of Motion Constraints [56.283944756315066]
  We propose an alternative TAMP approach that unifies task and motion planning into a single search. Our approach is based on an object-centric abstraction of motion constraints that permits leveraging the computational efficiency of off-the-shelf AI search to yield physically feasible plans.
  arXiv  Detail & Related papers  (2023-12-29T14:00:20Z)
• Interactive Planning Using Large Language Models for Partially Observable Robotics Tasks [54.60571399091711]
  Large Language Models (LLMs) have achieved impressive results in creating robotic agents for performing open vocabulary tasks. We present an interactive planning technique for partially observable tasks using LLMs.
  arXiv  Detail & Related papers  (2023-12-11T22:54:44Z)
• EgoPlan-Bench: Benchmarking Multimodal Large Language Models for Human-Level Planning [84.6451394629312]
  We introduce EgoPlan-Bench, a benchmark to evaluate the planning abilities of MLLMs in real-world scenarios. We show that EgoPlan-Bench poses significant challenges, highlighting a substantial scope for improvement in MLLMs to achieve human-level task planning. We also present EgoPlan-IT, a specialized instruction-tuning dataset that effectively enhances model performance on EgoPlan-Bench.
  arXiv  Detail & Related papers  (2023-12-11T03:35:58Z)
• Embodied Task Planning with Large Language Models [86.63533340293361]
  We propose a TAsk Planing Agent (TaPA) in embodied tasks for grounded planning with physical scene constraint. During inference, we discover the objects in the scene by extending open-vocabulary object detectors to multi-view RGB images collected in different achievable locations. Experimental results show that the generated plan from our TaPA framework can achieve higher success rate than LLaVA and GPT-3.5 by a sizable margin.
  arXiv  Detail & Related papers  (2023-07-04T17:58:25Z)
• Effective Baselines for Multiple Object Rearrangement Planning in Partially Observable Mapped Environments [5.32429768581469]
  This paper aims to enable home-assistive intelligent agents to efficiently plan for rearrangement under partial observability. We investigate monolithic and modular deep reinforcement learning (DRL) methods for planning in our setting. We find that monolithic DRL methods do not succeed at long-horizon planning needed for multi-object rearrangement. We also show that our greedy modular agents are empirically optimal when the objects that need to be rearranged are uniformly distributed in the environment.
  arXiv  Detail & Related papers  (2023-01-24T08:03:34Z)
• Sequence-Based Plan Feasibility Prediction for Efficient Task and Motion Planning [36.300564378022315]
  We present a learning-enabled Task and Motion Planning (TAMP) algorithm for solving mobile manipulation problems in environments with many articulated and movable obstacles. The core of our algorithm is PIGINet, a novel Transformer-based learning method that takes in a task plan, the goal, and the initial state, and predicts the probability of finding motion trajectories associated with the task plan.
  arXiv  Detail & Related papers  (2022-11-03T04:12:04Z)
• Learning to Search in Task and Motion Planning with Streams [20.003445874753233]
  Task and motion planning problems in robotics combine symbolic planning over discrete task variables with motion optimization over continuous state and action variables. We propose a geometrically informed symbolic planner that expands the set of objects and facts in a best-first manner. We apply our algorithm on a 7DOF robotic arm in block-stacking manipulation tasks.
  arXiv  Detail & Related papers  (2021-11-25T15:58:31Z)
• Task Allocation for Multi-Robot Task and Motion Planning: a case for Object Picking in Cluttered Workspaces [1.3535770763481902]
  We present an integrated multi-robot task and motion planning approach. It is capable of handling tasks which involve an unknown number of object re-arrangements. We demonstrate our results with experiments in simulation on two Franka Emika manipulators.
  arXiv  Detail & Related papers  (2021-10-08T12:36:43Z)
• Task Scoping: Generating Task-Specific Abstractions for Planning [19.411900372400183]
  Planning to solve any specific task using an open-scope world model is computationally intractable. We propose task scoping: a method that exploits knowledge of the initial condition, goal condition, and transition-dynamics structure of a task. We prove that task scoping never deletes relevant factors or actions, characterize its computational complexity, and characterize the planning problems for which it is especially useful.
  arXiv  Detail & Related papers  (2020-10-17T21:19:25Z)
• iCORPP: Interleaved Commonsense Reasoning and Probabilistic Planning on Robots [46.13039152809055]
  We present a novel algorithm, called iCORPP, to simultaneously estimate the current world state, reason about world dynamics, and construct task-oriented controllers. Results show significant improvements in scalability, efficiency, and adaptiveness, compared to competitive baselines.
  arXiv  Detail & Related papers  (2020-04-18T17:46:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.