Real-to-Sim for Highly Cluttered Environments via Physics-Consistent Inter-Object Reasoning

• URL: http://arxiv.org/abs/2602.12633v1
• Date: Fri, 13 Feb 2026 05:24:58 GMT
• Title: Real-to-Sim for Highly Cluttered Environments via Physics-Consistent Inter-Object Reasoning
• Abstract summary: Reconstructing physically valid 3D scenes from single-view observations is a prerequisite for bridging the gap between visual perception and robotic control.<n>We propose a novel physics-constrained Real-to-Sim pipeline that reconstructs physically consistent 3D scenes from single-view RGB-D data.
• Score: 11.05191602561784
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reconstructing physically valid 3D scenes from single-view observations is a prerequisite for bridging the gap between visual perception and robotic control. However, in scenarios requiring precise contact reasoning, such as robotic manipulation in highly cluttered environments, geometric fidelity alone is insufficient. Standard perception pipelines often neglect physical constraints, resulting in invalid states, e.g., floating objects or severe inter-penetration, rendering downstream simulation unreliable. To address these limitations, we propose a novel physics-constrained Real-to-Sim pipeline that reconstructs physically consistent 3D scenes from single-view RGB-D data. Central to our approach is a differentiable optimization pipeline that explicitly models spatial dependencies via a contact graph, jointly refining object poses and physical properties through differentiable rigid-body simulation. Extensive evaluations in both simulation and real-world settings demonstrate that our reconstructed scenes achieve high physical fidelity and faithfully replicate real-world contact dynamics, enabling stable and reliable contact-rich manipulation.

Related papers

• GARDEN: Gravity-Aligned Reconstruction of Disentangled ENvironments from RGB images [61.2170105836525]
  We propose an RGB-only framework that reformulates reconstruction as physically-grounded scene factorization and outputs a structured hybrid scene representation.<n>The resulting representation combines explicit rigid bodies with a decoupled background, enabling direct physics simulation while preserving visual realism.
  arXiv  Detail & Related papers  (2026-06-02T17:13:01Z)
• Automatically Improving Simulation Physics for Articulated Objects [0.0]
  We introduce interaction-readiness, which characterizes whether an object can be reliably simulated under manipulation.<n>We present a simulator-in-the-loop approach for generating interaction-ready articulated objects from incomplete 3D assets.
  arXiv  Detail & Related papers  (2026-05-18T21:34:33Z)
• Physically Grounded 3D Generative Reconstruction under Hand Occlusion using Proprioception and Multi-Contact Touch [3.926587614210279]
  We propose a multimodal, physically grounded approach for metric-scale amodal object reconstruction and pose estimation under hand occlusion.<n>We leverage physical interaction signals: proprioception provides the posed hand geometry, and multi-contact touch constrains where the object surface must lie.
  arXiv  Detail & Related papers  (2026-04-10T08:32:51Z)
• Resonance4D: Frequency-Domain Motion Supervision for Preset-Free Physical Parameter Learning in 4D Dynamic Physical Scene Simulation [53.141181758015186]
  We present Resonance4D, a physics-driven 4D dynamic simulation framework that couples 3D Gaussian Splatting with the Material Point Method.<n>Our key insight is that dynamic consistency can be enforced without dense temporal generation by jointly constraining motion in complementary domains.<n>We show that Resonance4D achieves strong physical fidelity and motion consistency while reducing peak GPU memory from over 35,GB to around 20,GB.
  arXiv  Detail & Related papers  (2026-04-02T13:00:22Z)
• FluidGaussian: Propagating Simulation-Based Uncertainty Toward Functionally-Intelligent 3D Reconstruction [37.29218600173778]
  We propose a plug-and-play method that tightly couples geometry reconstruction with ubiquitous fluid-structure interactions to assess surface quality at high granularity.<n>Our FluidGaussian method yields up to +8.6% visual PSNR (Peak Signal-to-Noise Ratio) and -62.3% velocity divergence during fluid simulations.
  arXiv  Detail & Related papers  (2026-03-22T18:30:57Z)
• HSImul3R: Physics-in-the-Loop Reconstruction of Simulation-Ready Human-Scene Interactions [69.43316445143408]
  We present HSImul3R, a unified framework for simulation-ready 3D reconstruction of human-scene interactions.<n>Existing methods suffer from a perception-simulation gap, leading to instability in physics engines and failure in embodied AI applications.
  arXiv  Detail & Related papers  (2026-03-16T17:58:33Z)
• URDF-Anything+: Autoregressive Articulated 3D Models Generation for Physical Simulation [45.4820195450296]
  We present an end-to-end autoregressive framework that directly generates executable articulated object models from visual observations.<n>Building on this capability, we enable a new Real-Follow-Sim paradigm, where high-fidelity digital twins constructed from visual observations allow policies trained in simulation to transfer to real robots without online adaptation.
  arXiv  Detail & Related papers  (2026-03-14T16:23:44Z)
• MotionAnymesh: Physics-Grounded Articulation for Simulation-Ready Digital Twins [25.25391483526521]
  MotionAnymesh is a framework that seamlessly transforms unstructured static meshes into simulation-ready digital twins.<n>Our method features a kinematic-aware part segmentation module that grounds VLM reasoning with explicit SP4D physical priors.
  arXiv  Detail & Related papers  (2026-03-13T12:30:42Z)
• PhyScensis: Physics-Augmented LLM Agents for Complex Physical Scene Arrangement [89.35154754765502]
  PhyScensis is an agent-based framework powered by a physics engine to produce physically plausible scene configurations.<n>Our framework preserves strong controllability over fine-grained textual descriptions and numerical parameters.<n> Experimental results show that our method outperforms prior approaches in scene complexity, visual quality, and physical accuracy.
  arXiv  Detail & Related papers  (2026-02-16T17:55:25Z)
• Mirage2Matter: A Physically Grounded Gaussian World Model from Video [87.9732484393686]
  We present Simulate Anything, a graphics-driven world modeling and simulation framework.<n>Our approach reconstructs real-world environments into a photorealistic scene representation using 3D Gaussian Splatting (3DGS)<n>We then leverage generative models to recover a physically realistic representation and integrate it into a simulation environment via a precision calibration target.
  arXiv  Detail & Related papers  (2026-01-24T07:43:57Z)
• DeepSimHO: Stable Pose Estimation for Hand-Object Interaction via Physics Simulation [81.11585774044848]
  We present DeepSimHO, a novel deep-learning pipeline that combines forward physics simulation and backward gradient approximation with a neural network. Our method noticeably improves the stability of the estimation and achieves superior efficiency over test-time optimization.
  arXiv  Detail & Related papers  (2023-10-11T05:34:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.