Perceive-then-Plan: Layout-as-Policy for Monocular 3D Scene Layout Estimation

• URL: http://arxiv.org/abs/2605.25326v1
• Date: Mon, 25 May 2026 01:16:19 GMT
• Title: Perceive-then-Plan: Layout-as-Policy for Monocular 3D Scene Layout Estimation
• Abstract summary: Building structured 3D scene layouts from a single image requires reconciling visual observations with physical and spatial constraints.<n>We formulate monocular 3D layout estimation as a perceive-then-plan problem with vision-language models.<n>We propose layout-as-Policy (LaP), which casts the planning stage as a policy learning problem.
• Score: 37.60004902691764
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Building structured 3D scene layouts from a single image requires reconciling visual observations with physical and spatial constraints, a challenge that is difficult to address with direct prediction alone. In this work, we formulate monocular 3D layout estimation as a perceive-then-plan problem with vision-language models, where a Perceiver first grounds the 3D objects and then a Planner iteratively refines the scene hypothesis through actions that improve physical plausibility while preserving consistency with the input image. We propose Layout-as-Policy (LaP), which casts the planning stage as a policy learning problem: 3D layouts are represented as structured states, and refined via discrete actions such as translation, rotation, and rescaling. Starting from an observation-aligned initialization with the geometry-enhanced Perceiver, the LaP Planner is trained to produce action sequences that progressively resolve geometric inconsistencies and enforce realistic spatial relations. To enable effective learning, we combine supervised trajectory initialization with preference-based optimization, allowing the model to learn corrective behaviors without requiring explicit reward engineering. This formulation transforms layout estimation from a one-shot prediction task into an iterative refinement process, enabling better handling of global constraints and complex object interactions. Experiments demonstrate that our approach produces layouts that are more physically coherent and better aligned with visual observations, while naturally supporting downstream tasks such as scene editing and manipulation.

Related papers

• SceneConductor: 3D Scene Generation from a Single Image with Multi-Agent Orchestration [32.39337008619354]
  We propose a multi-agent orchestration framework that decomposes single-image 3D scene generation into three structured stages.<n>We introduce a geometry-aware layout predictor supervised by sparse geometric priors derived from point maps.<n>Our method consistently outperforms prior approaches in geometric accuracy, spatial consistency, and perceptual realism.
  arXiv  Detail & Related papers  (2026-06-07T01:38:39Z)
• Z-FLoc: Zero-Shot Floorplan Localization via Geometric Primitives [72.63181031215858]
  We propose a zero-shot floorplan localization method that generalizes to novel environments without any retraining.<n>Our key insight is that dominant geometric primitives are ubiquitous in human-made environments.<n> Experiments on both simulated and real-world datasets show that our approach outperforms state-of-the-art learning-based methods on unseen environments.
  arXiv  Detail & Related papers  (2026-06-03T12:14:24Z)
• STABLE: Simulation-Ready Tabletop Layout Generation via a Semantics-Physics Dual System [76.3920413336862]
  We present STABLE, a semantics-physics dual-system tailored for simulation-ready tabletop scene generation.<n>STABLE consists of two complementary modules: a Semantic Reasoner and a Physics Corrector.<n> Experiments demonstrate that STABLE successfully generates simulation-ready tabletop scenes that strictly conform to task instructions.
  arXiv  Detail & Related papers  (2026-05-15T16:18:42Z)
• Perspective from a Higher Dimension: Can 3D Geometric Priors Help Visual Floorplan Localization? [8.82283453148819]
  Self-localization of building's floorplans has attracted researchers' interest.<n>Since floorplans are minimalist representations of a building's structure, modal and geometric differences between visual perceptions and floorplans pose challenges to this task.<n>Existing methods cleverly utilize 2D geometric features and pose filters to achieve promising performance.<n>This paper views the 2D Floorplan localization problem from a higher dimension by injecting 3D geometric priors into the visual FLoc algorithm.
  arXiv  Detail & Related papers  (2025-07-25T01:34:26Z)
• Unposed Sparse Views Room Layout Reconstruction in the Age of Pretrain Model [15.892685514932323]
  We introduce Plane-DUSt3R, a novel method for multi-view room layout estimation.<n>Plane-DUSt3R incorporates the DUSt3R framework and fine-tunes on a room layout dataset (Structure3D) with a modified objective to estimate structural planes.<n>By generating uniform and parsimonious results, Plane-DUSt3R enables room layout estimation with only a single post-processing step and 2D detection results.
  arXiv  Detail & Related papers  (2025-02-24T02:14:19Z)
• MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion [118.74385965694694]
  We present Motion DUSt3R (MonST3R), a novel geometry-first approach that directly estimates per-timestep geometry from dynamic scenes.<n>By simply estimating a pointmap for each timestep, we can effectively adapt DUST3R's representation, previously only used for static scenes, to dynamic scenes.<n>We show that by posing the problem as a fine-tuning task, identifying several suitable datasets, and strategically training the model on this limited data, we can surprisingly enable the model to handle dynamics.
  arXiv  Detail & Related papers  (2024-10-04T18:00:07Z)
• Exploiting Priors from 3D Diffusion Models for RGB-Based One-Shot View Planning [24.44253219419552]
  We propose a novel one-shot view planning approach that utilizes the powerful 3D generation capabilities of diffusion models as priors. Our experiments in simulation and real-world setups indicate that our approach balances well between object reconstruction quality and movement cost.
  arXiv  Detail & Related papers  (2024-03-25T14:21:49Z)
• Reconstructing Interactive 3D Scenes by Panoptic Mapping and CAD Model Alignments [81.38641691636847]
  We rethink the problem of scene reconstruction from an embodied agent's perspective. We reconstruct an interactive scene using RGB-D data stream. This reconstructed scene replaces the object meshes in the dense panoptic map with part-based articulated CAD models.
  arXiv  Detail & Related papers  (2021-03-30T05:56:58Z)
• SCFusion: Real-time Incremental Scene Reconstruction with Semantic Completion [86.77318031029404]
  We propose a framework that performs scene reconstruction and semantic scene completion jointly in an incremental and real-time manner. Our framework relies on a novel neural architecture designed to process occupancy maps and leverages voxel states to accurately and efficiently fuse semantic completion with the 3D global model.
  arXiv  Detail & Related papers  (2020-10-26T15:31:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.