REST3D: Reconstructing Physically Stable 3D Scenes from a Single Image

• URL: http://arxiv.org/abs/2605.30338v1
• Date: Thu, 28 May 2026 17:59:01 GMT
• Title: REST3D: Reconstructing Physically Stable 3D Scenes from a Single Image
• Abstract summary: Reconstructing physically stable 3D scenes from a single RGB image enables casual images to be converted into simulation-ready digital assets.<n>We propose REST3D, a single-image reconstruction framework that can reconstruct physically stable 3D scenes by integrating physical scene understanding with physics-constrained refinement.
• Score: 31.061246129846044
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Reconstructing physically stable 3D scenes from a single RGB image enables casual images to be converted into simulation-ready digital assets for applications such as immersive interaction and content creation. However, existing single-image reconstruction methods fall short in capturing the physical structure of a scene. As a result, they often produce geometrically plausible but physically inconsistent results, including object floating and penetration, which lead to unstable behavior in physics simulations. Image-conditioned scene generation methods improve physical plausibility but often rely on strong scene priors, yielding plausible yet inaccurate object arrangements that fail to match the input image. We propose REST3D, a single-image reconstruction framework that can reconstruct physically stable 3D scenes by integrating physical scene understanding with physics-constrained refinement. We first introduce an agentic physical scene understanding technique that constructs a scene-tree representation capturing object physical states and inter-object relationships from a gravity-support perspective, providing a structural prior for reconstruction. Leveraging this structure, we initialize the scene using image-to-3D models, followed by scene-tree-guided alignment and physics-constrained optimization to resolve physical violations while preserving visual consistency with the input image. Experiments show that our method significantly reduces physical errors and improves simulation stability on both synthetic and real-world datasets while maintaining strong reconstruction quality. We further demonstrate the reconstructed scenes in VR-based human-object interaction, showing their potential for immersive applications.

Related papers

• SimWorlds: A Multi-Agent System for Dynamic 3D Scene Creation [50.175987640001644]
  We present SimWorlds: a multi-agent framework that produces dynamic, editable 4D scenes from text.<n>We also introduce 4DBuildBench, a benchmark for assessing both visual fidelity and physical consistency of the procedural dynamic 3D scenes.
  arXiv  Detail & Related papers  (2026-07-02T06:29:18Z)
• $φ$-Scene: Physically Grounded Image-to-3D Scene Reconstruction [48.26103456970799]
  $$-Scene is a physically grounded approach to open-vocabulary and compositional image-to-3D scene reconstruction.<n>It infers how objects support one another, orders them accordingly, and settles each object against its stabilized support context.<n>Experiments on 3D-Front show that $$-Scene achieves the strongest overall performance among out-of-domain methods.
  arXiv  Detail & Related papers  (2026-06-19T16:50:48Z)
• SimuScene: Simulation-Ready Compositional 3D Scene Reconstruction from a Single Image [18.870075620011615]
  We introduce SimuScene, a compositional 3D reconstruction pipeline that puts physics in the loop of shape and layout estimation.<n>By diagnostically simulating reconstructed objects under gravity, we convert penetration and support failures into quantitative correction signals.<n>This physics-informed feedback loop mitigates accumulated reconstruction errors and produces a stable, simulation-ready scene.
  arXiv  Detail & Related papers  (2026-06-02T17:59:59Z)
• GARDEN: Gravity-Aligned Reconstruction of Disentangled ENvironments from RGB images [61.2170105836525]
  We propose an RGB-only framework that reformulates reconstruction as physically-grounded scene factorization and outputs a structured hybrid scene representation.<n>The resulting representation combines explicit rigid bodies with a decoupled background, enabling direct physics simulation while preserving visual realism.
  arXiv  Detail & Related papers  (2026-06-02T17:13:01Z)
• SAM3D-Phys: Towards Multi-Object Interactive Simulation in Real World [42.46212677643043]
  SAM3D-Phys is a framework that integrates scene reconstruction with generative 3D priors of SAM3D to recover physically simulatable objects.<n>By recovering complete object geometry and restoring its pose and appearance within the scene, SAM3D-Phys produces clean object representations suitable for physics-based simulation.
  arXiv  Detail & Related papers  (2026-05-28T17:07:51Z)
• TelePhysics: Physics-Grounded Multi-Object Scene Generation from a Single Image with Real-Time Interaction [51.01447538245441]
  We propose TelePhysics, a training-free framework that converts a single image into a physically consistent and controllable video.<n>By representing the full scene geometry in a unified spatial coordinate system, TelePhysics resolves object penetration and alignment ambiguity.<n> Experimental results demonstrate that TelePhysics substantially outperforms prior methods in physical fidelity, spatial coherence, and controllability.
  arXiv  Detail & Related papers  (2026-05-19T08:16:44Z)
• PAT3D: Physics-Augmented Text-to-3D Scene Generation [47.18949891825537]
  PAT3D generates 3D objects, infers their spatial relations, and organizes them into a hierarchical scene tree.<n>A differentiable rigid-body simulator ensures realistic object interactions under gravity.<n>Experiments demonstrate that PAT3D substantially outperforms prior approaches in physical plausibility, semantic consistency, and visual quality.
  arXiv  Detail & Related papers  (2025-11-26T23:23:58Z)
• Physically Compatible 3D Object Modeling from a Single Image [109.98124149566927]
  We present a framework that transforms single images into 3D physical objects.<n>Our framework embeds physical compatibility into the reconstruction process.<n>It consistently enhances the physical realism of 3D models over existing methods.
  arXiv  Detail & Related papers  (2024-05-30T21:59:29Z)
• Reconstructing Interactive 3D Scenes by Panoptic Mapping and CAD Model Alignments [81.38641691636847]
  We rethink the problem of scene reconstruction from an embodied agent's perspective. We reconstruct an interactive scene using RGB-D data stream. This reconstructed scene replaces the object meshes in the dense panoptic map with part-based articulated CAD models.
  arXiv  Detail & Related papers  (2021-03-30T05:56:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.