Embodied Scene Rearrangement Planning

• URL: http://arxiv.org/abs/2608.27371v1
• Date: Thu, 27 Aug 2026 17:08:40 GMT
• Title: Embodied Scene Rearrangement Planning
• Abstract summary: This paper introduces Embodied Scene Rearrangement Planning (ESRP)<n>ESRP requires embodied agents to rearrange furniture in 3D scenes to match a target configuration using only egocentric observations and a top-down target layout.<n>We present ESRP-Bench, a comprehensive benchmark built on OmniGibson featuring over 5,400 scene pairs and 8,200 objects.
• Score: 30.63533710397442
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper introduces Embodied Scene Rearrangement Planning (ESRP), a novel task requiring embodied agents to rearrange furniture in 3D scenes to match a target configuration using only egocentric observations and a top-down target layout. Unlike prior rearrangement tasks, ESRP precludes global state access and introduces mutual object occlusions, reflecting the practical constraints of real-world robotic deployment. These factors make aligning partial egocentric observations with the global target layout particularly challenging for long-horizon planning. To facilitate research, we present ESRP-Bench, a comprehensive benchmark built on OmniGibson featuring over 5,400 scene pairs and 8,200 objects. We define three multi-level metrics to evaluate rearrangement quality and provide four baselines: a hierarchical task-and-motion planning method, a vision-language-model-based method, and two learning-based approaches (IL and RL). Experimental results demonstrate that current methods struggle to complete the task efficiently, highlighting ESRP as a challenging frontier for embodied agents in scene understanding and long-horizon task planning. This work serves as a stepping stone toward deploying intelligent agents in real-world scenarios. Project page: https://pie-lab.cn/ESRP/.

Related papers

• World Action Planner: Generalizable Decision-Making with Action-Conditioned World Models [54.68021681438324]
  We propose World Action Planner, a robot planning system that leverages the reasoning capabilities of Vision-Language Models (VLMs)<n>Our system enables an agent to propose initial action plans and iteratively refine them via optimization and search, reasoning over imagined world model rollouts.<n>We demonstrate that our approach achieves superior performance across compositional tasks, new layouts, and zero-shot generalization scenarios.
  arXiv  Detail & Related papers  (2026-07-30T02:41:52Z)
• Hierarchical 3D Scene Graph Construction and Belief-based Planning for Semantic Navigation [34.08516675160383]
  We propose a zero-shot semantic navigation framework for embodied agents.<n>Our method incrementally maintains an online Hierarchical 3D Scene Graph (HSG) over objects, zones, and regions, serving as a compact state abstraction for global planning.<n>Building on this memory, we introduce a hierarchical belief-based planning framework that fuses semantic priors with exploration evidence on the HSG, and performs finite-horizon rollouts on an HSG-based simulator to explicitly estimate the long-term expected returns of candidate macro-actions.
  arXiv  Detail & Related papers  (2026-06-30T03:01:44Z)
• RePlan-Bot: Multi-Level Replanning for Embodied Instruction Following [39.9780803736704]
  Embodied instruction following (EIF) requires agents to understand and execute complex natural language commands within interactive 3D environments.<n>We introduce RePlan-Bot, a novel EIF agent that performs multi-level, continuous replanning throughout task execution.
  arXiv  Detail & Related papers  (2026-05-25T13:44:18Z)
• Spatially Grounded Long-Horizon Task Planning in the Wild [49.614308474374035]
  GroundedPlanBench is a newly curated benchmark for spatially grounded long-horizon action planning in the wild.<n>Video-to-Spatially Grounded Planning (V2GP) is an automated data generation framework that leverages real-world robot video demonstrations to improve spatially grounded long-horizon planning.
  arXiv  Detail & Related papers  (2026-03-13T05:42:18Z)
• MaP-AVR: A Meta-Action Planner for Agents Leveraging Vision Language Models and Retrieval-Augmented Generation [18.84633713315585]
  Embodied robotic AI systems designed to manage complex daily tasks rely on a task planner to understand and decompose high-level tasks.<n>This paper argues that defining the planned skill set is equally crucial.<n>To handle the complexity of daily environments, the skill set should possess a high degree of generalization ability.
  arXiv  Detail & Related papers  (2025-12-22T14:58:52Z)
• Hierarchical Object-Oriented POMDP Planning for Object Rearrangement [19.62753215239688]
  Current object rearrangement solutions, primarily based on Reinforcement Learning or hand-coded planning methods, often lack adaptability to diverse challenges.<n>To address this limitation, we introduce a novel Hierarchical Object-Oriented Partially Observed Markov Decision Process (HOO-POMDP) planning approach.<n>We present an online planning framework and a new benchmark dataset for solving multi-object rearrangement problems in partially observable, multi-room environments.
  arXiv  Detail & Related papers  (2024-12-02T10:19:36Z)
• Affordances-Oriented Planning using Foundation Models for Continuous Vision-Language Navigation [64.84996994779443]
  We propose a novel Affordances-Oriented Planner for continuous vision-language navigation (VLN) task. Our AO-Planner integrates various foundation models to achieve affordances-oriented low-level motion planning and high-level decision-making. Experiments on the challenging R2R-CE and RxR-CE datasets show that AO-Planner achieves state-of-the-art zero-shot performance.
  arXiv  Detail & Related papers  (2024-07-08T12:52:46Z)
• Embodied Instruction Following in Unknown Environments [64.57388036567461]
  We propose an embodied instruction following (EIF) method for complex tasks in the unknown environment.<n>We build a hierarchical embodied instruction following framework including the high-level task planner and the low-level exploration controller.<n>For the task planner, we generate the feasible step-by-step plans for human goal accomplishment according to the task completion process and the known visual clues.
  arXiv  Detail & Related papers  (2024-06-17T17:55:40Z)
• Learning adaptive planning representations with natural language guidance [90.24449752926866]
  This paper describes Ada, a framework for automatically constructing task-specific planning representations. Ada interactively learns a library of planner-compatible high-level action abstractions and low-level controllers adapted to a particular domain of planning tasks.
  arXiv  Detail & Related papers  (2023-12-13T23:35:31Z)
• Embodied Task Planning with Large Language Models [86.63533340293361]
  We propose a TAsk Planing Agent (TaPA) in embodied tasks for grounded planning with physical scene constraint. During inference, we discover the objects in the scene by extending open-vocabulary object detectors to multi-view RGB images collected in different achievable locations. Experimental results show that the generated plan from our TaPA framework can achieve higher success rate than LLaVA and GPT-3.5 by a sizable margin.
  arXiv  Detail & Related papers  (2023-07-04T17:58:25Z)
• Enabling Visual Action Planning for Object Manipulation through Latent Space Roadmap [72.01609575400498]
  We present a framework for visual action planning of complex manipulation tasks with high-dimensional state spaces. We propose a Latent Space Roadmap (LSR) for task planning, a graph-based structure capturing globally the system dynamics in a low-dimensional latent space. We present a thorough investigation of our framework on two simulated box stacking tasks and a folding task executed on a real robot.
  arXiv  Detail & Related papers  (2021-03-03T17:48:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.