Spatially Grounded Long-Horizon Task Planning in the Wild

• URL: http://arxiv.org/abs/2603.13433v1
• Date: Fri, 13 Mar 2026 05:42:18 GMT
• Title: Spatially Grounded Long-Horizon Task Planning in the Wild
• Abstract summary: GroundedPlanBench is a newly curated benchmark for spatially grounded long-horizon action planning in the wild.<n>Video-to-Spatially Grounded Planning (V2GP) is an automated data generation framework that leverages real-world robot video demonstrations to improve spatially grounded long-horizon planning.
• Score: 49.614308474374035
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in robot manipulation increasingly leverage Vision-Language Models (VLMs) for high-level reasoning, such as decomposing task instructions into sequential action plans expressed in natural language that guide downstream low-level motor execution. However, current benchmarks do not assess whether these plans are spatially executable, particularly in specifying the exact spatial locations where the robot should interact to execute the plan, limiting evaluation of real-world manipulation capability. To bridge this gap, we define a novel task of grounded planning and introduce GroundedPlanBench, a newly curated benchmark for spatially grounded long-horizon action planning in the wild. GroundedPlanBench jointly evaluates hierarchical sub-action planning and spatial action grounding (where to act), enabling systematic assessment of whether generated sub-actions are spatially executable for robot manipulation. We further introduce Video-to-Spatially Grounded Planning (V2GP), an automated data generation framework that leverages real-world robot video demonstrations to improve spatially grounded long-horizon planning. Our evaluations reveal that spatially grounded long-horizon planning remains a major bottleneck for current VLMs. Our results demonstrate that V2GP provides a promising approach for improving both action planning and spatial grounding performance, validated on our benchmark as well as through real-world robot manipulation experiments, advancing progress toward spatially actionable planning.

Related papers

• VIP: Variation-based Iterative-learning Planning for Robotic Navigation [14.214543473511048]
  This paper presents a novel variation-based iterative-learning planning framework for efficient motion planning of robots.<n>The proposed framework can efficiently generate and iteratively improve motion plans for different planning objectives, robotic platforms, and swarm configurations.
  arXiv  Detail & Related papers  (2026-08-25T14:38:20Z)
• Simulation-Informed Diffusion for Decentralized Multi-robot Motion Planning [56.240199425429445]
  Multi-robot motion planning requires each robot to generate collision-free trajectories from local observations.<n>Most existing planners generate trajectories from a static snapshot of the local observation.<n>This paper introduces Simulation-Informed Diffusion (SID), a decentralized framework built on constraint-aware diffusion models.
  arXiv  Detail & Related papers  (2026-05-26T21:17:53Z)
• Navigating the Clutter: Waypoint-Based Bi-Level Planning for Multi-Robot Systems [54.887871365121775]
  Multi-robot control in cluttered environments is a challenging problem that involves complex physical constraints.<n>We propose a hybrid multi-robot control framework that jointly optimize task and motion planning.<n>Our approach consistently improves task success over motion-agnostic and VLA-based baselines.
  arXiv  Detail & Related papers  (2026-04-22T22:58:47Z)
• Plan-MCTS: Plan Exploration for Action Exploitation in Web Navigation [50.406803870992974]
  Plan-MCTS is a framework that reformulates web navigation by shifting exploration to a semantic Plan Space.<n>Plan-MCTS achieves state-of-the-art performance, surpassing current approaches with higher task effectiveness and search efficiency.
  arXiv  Detail & Related papers  (2026-02-15T10:24:45Z)
• Grounding Generated Videos in Feasible Plans via World Models [52.63206803295352]
  Grounding Video Plans with World Models (GVP-WM) is a planning method that grounds video-generated plans into feasible action sequences.<n>GVP-WM first generates a video plan from initial and goal observations, then projects the video guidance onto the manifold of dynamically feasible latent trajectories.
  arXiv  Detail & Related papers  (2026-02-02T11:04:47Z)
• Spatial Policy: Guiding Visuomotor Robotic Manipulation with Spatial-Aware Modeling and Reasoning [43.35947581429867]
  Vision-centric hierarchical embodied models have demonstrated strong potential for long-horizon robotic control.<n>Existing methods lack spatial awareness capabilities, limiting their effectiveness in bridging visual plans to actionable control in complex environments.<n>We propose Spatial Policy (SP), a unified spatial-aware visuomotor robotic manipulation framework via explicit spatial modeling and reasoning.
  arXiv  Detail & Related papers  (2025-08-21T10:24:18Z)
• Gondola: Grounded Vision Language Planning for Generalizable Robotic Manipulation [62.711546725154314]
  We introduce Gondola, a grounded vision-language planning model based on large language models (LLMs) for generalizable robotic manipulation.<n>G Gondola takes multi-view images and history plans to produce the next action plan with interleaved texts and segmentation masks of target objects and locations.<n>G Gondola outperforms the state-of-the-art LLM-based method across all four levels of the GemBench dataset.
  arXiv  Detail & Related papers  (2025-06-12T20:04:31Z)
• Efficient Robotic Policy Learning via Latent Space Backward Planning [17.770562202624962]
  Current robotic planning methods often rely on predicting multi-frame images with full pixel details.<n>We propose a Latent Space Backward Planning scheme (LBP), which begins by grounding the task into final latent goals.<n>We show that LBP outperforms existing fine-grained and forward planning methods, achieving SOTA performance.
  arXiv  Detail & Related papers  (2025-05-11T06:13:51Z)
• Learning to Reason over Scene Graphs: A Case Study of Finetuning GPT-2 into a Robot Language Model for Grounded Task Planning [45.51792981370957]
  We investigate the applicability of a smaller class of large language models (LLMs) in robotic task planning by learning to decompose tasks into subgoal specifications for a planner to execute sequentially. Our method grounds the input of the LLM on the domain that is represented as a scene graph, enabling it to translate human requests into executable robot plans. Our findings suggest that the knowledge stored in an LLM can be effectively grounded to perform long-horizon task planning, demonstrating the promising potential for the future application of neuro-symbolic planning methods in robotics.
  arXiv  Detail & Related papers  (2023-05-12T18:14:32Z)
• Leveraging Scene Embeddings for Gradient-Based Motion Planning in Latent Space [24.95320093765214]
  AMP-LS is able to plan in novel, complex scenes while outperforming traditional planning baselines in terms of speed by an order of magnitude. We show that the resulting system is fast enough to enable closed-loop planning in real-world dynamic scenes.
  arXiv  Detail & Related papers  (2023-03-06T18:49:39Z)
• Large-scale Autonomous Flight with Real-time Semantic SLAM under Dense Forest Canopy [48.51396198176273]
  We propose an integrated system that can perform large-scale autonomous flights and real-time semantic mapping in challenging under-canopy environments. We detect and model tree trunks and ground planes from LiDAR data, which are associated across scans and used to constrain robot poses as well as tree trunk models. A drift-compensation mechanism is designed to minimize the odometry drift using semantic SLAM outputs in real time, while maintaining planner optimality and controller stability.
  arXiv  Detail & Related papers  (2021-09-14T07:24:53Z)
• Latent Space Roadmap for Visual Action Planning of Deformable and Rigid Object Manipulation [74.88956115580388]
  Planning is performed in a low-dimensional latent state space that embeds images. Our framework consists of two main components: a Visual Foresight Module (VFM) that generates a visual plan as a sequence of images, and an Action Proposal Network (APN) that predicts the actions between them.
  arXiv  Detail & Related papers  (2020-03-19T18:43:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.