Real2Sim based on Active Perception with automatically VLM-generated Behavior Trees

• URL: http://arxiv.org/abs/2601.08454v1
• Date: Tue, 13 Jan 2026 11:28:46 GMT
• Title: Real2Sim based on Active Perception with automatically VLM-generated Behavior Trees
• Abstract summary: This paper presents a Real2Sim framework that autonomously generates and executes Behavior Trees for task-specific physical interactions.<n>Given a high-level user request, an incomplete simulation description, and an RGB observation of the scene, a vision-language model performs multi-modal reasoning to identify relevant objects.<n>The resulting behavior is executed on a torque-controlled Franka Emika Panda, enabling compliant, contact-rich interactions for parameter estimation.
• Score: 41.21672421530866
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Constructing an accurate simulation model of real-world environments requires reliable estimation of physical parameters such as mass, geometry, friction, and contact surfaces. Traditional real-to-simulation (Real2Sim) pipelines rely on manual measurements or fixed, pre-programmed exploration routines, which limit their adaptability to varying tasks and user intents. This paper presents a Real2Sim framework that autonomously generates and executes Behavior Trees for task-specific physical interactions to acquire only the parameters required for a given simulation objective, without relying on pre-defined task templates or expert-designed exploration routines. Given a high-level user request, an incomplete simulation description, and an RGB observation of the scene, a vision-language model performs multi-modal reasoning to identify relevant objects, infer required physical parameters, and generate a structured Behavior Tree composed of elementary robotic actions. The resulting behavior is executed on a torque-controlled Franka Emika Panda, enabling compliant, contact-rich interactions for parameter estimation. The acquired measurements are used to automatically construct a physics-aware simulation. Experimental results on the real manipulator demonstrate estimation of object mass, surface height, and friction-related quantities across multiple scenarios, including occluded objects and incomplete prior models. The proposed approach enables interpretable, intent-driven, and autonomously Real2Sim pipelines, bridging high-level reasoning with physically-grounded robotic interaction.

Related papers

• WorldSimProbe: Diagnosing Simulator Faithfulness in Action-Conditioned World Models for Embodied Manipulation [85.15806404962193]
  Action-conditioned world models (ACWMs) promise to provide embodied AI with scalable predictive simulators for planning, policy evaluation, and data generation.<n>We evaluate ACWMs through the observable capabilities expected of physical simulators.<n>WorldSimProbe reveals systematic action-realization degradation across control variation, structured failures in interaction grounding and dynamics.
  arXiv  Detail & Related papers  (2026-08-10T08:48:06Z)
• Agentic Real2Sim: Physics-based World Modeling with Vision-Language Agents [42.18672329354181]
  We introduce textitAgentic Real2Sim, a framework for generalized physical world modeling with vision-language agents.<n>We evaluate Agentic Real2Sim on rigid-object manipulation, deformable-object interaction, and humanoid motion scenes.<n>The framework's agentic decisions can be driven by an open-weight VLM backend at a small fraction of the cost of frontier models.
  arXiv  Detail & Related papers  (2026-07-21T15:23:38Z)
• RigPI: Dynamic Parameter Identification of Rigid Body via VLM-Seeded Differentiable Simulation [16.686983429272214]
  RigPI is a framework for identifying dynamic parameters of rigid bodies and multi-link rigid bodies during robot-object interaction.<n> RigPI integrates vision-based semantic priors, force-torque measurements, and motion observations within a differentiable simulation pipeline.<n>Extensive real-world experiments on objects with revolute and prismatic joints demonstrate that RigPI achieves accurate and stable parameter estimates.
  arXiv  Detail & Related papers  (2026-06-23T22:15:42Z)
• Automatically Improving Simulation Physics for Articulated Objects [0.0]
  We introduce interaction-readiness, which characterizes whether an object can be reliably simulated under manipulation.<n>We present a simulator-in-the-loop approach for generating interaction-ready articulated objects from incomplete 3D assets.
  arXiv  Detail & Related papers  (2026-05-18T21:34:33Z)
• URDF-Anything+: Autoregressive Articulated 3D Models Generation for Physical Simulation [45.4820195450296]
  We present an end-to-end autoregressive framework that directly generates executable articulated object models from visual observations.<n>Building on this capability, we enable a new Real-Follow-Sim paradigm, where high-fidelity digital twins constructed from visual observations allow policies trained in simulation to transfer to real robots without online adaptation.
  arXiv  Detail & Related papers  (2026-03-14T16:23:44Z)
• Real-to-Sim for Highly Cluttered Environments via Physics-Consistent Inter-Object Reasoning [11.05191602561784]
  Reconstructing physically valid 3D scenes from single-view observations is a prerequisite for bridging the gap between visual perception and robotic control.<n>We propose a novel physics-constrained Real-to-Sim pipeline that reconstructs physically consistent 3D scenes from single-view RGB-D data.
  arXiv  Detail & Related papers  (2026-02-13T05:24:58Z)
• Contact-Aware Neural Dynamics [42.83872574010789]
  We propose an implicit sim-to-real framework that learns to align the simulator's dynamics with contact information.<n>We show that using tactile contact information from robotic hands can effectively model the non-smooth discontinuities inherent in contact-rich tasks.
  arXiv  Detail & Related papers  (2026-01-19T07:56:13Z)
• SIMPACT: Simulation-Enabled Action Planning using Vision-Language Models [60.80050275581661]
  Vision-Language Models (VLMs) exhibit remarkable common-sense and semantic reasoning capabilities.<n>They lack a grounded understanding of physical dynamics.<n>We present S, a test-time, SIMulation-enabled ACTion Planning framework.<n>Our method demonstrates state-of-the-art performance on five challenging, real-world rigid-body and deformable manipulation tasks.
  arXiv  Detail & Related papers  (2025-12-05T18:51:03Z)
• Kinematify: Open-Vocabulary Synthesis of High-DoF Articulated Objects [59.51185639557874]
  We introduce Kinematify, an automated framework that synthesizes articulated objects directly from arbitrary RGB images or textual descriptions.<n>Our method addresses two core challenges: (i) inferring kinematic topologies for high-DoF objects and (ii) estimating joint parameters from static geometry.
  arXiv  Detail & Related papers  (2025-11-03T07:21:42Z)
• URDF-Anything: Constructing Articulated Objects with 3D Multimodal Language Model [76.08429266631823]
  We propose an end-to-end automatic reconstruction framework based on a 3D multimodal large language model (MLLM)<n>URDF-Anything utilizes an autoregressive prediction framework based on point-cloud and text multimodal input to jointly optimize geometric segmentation and kinematic parameter prediction.<n> Experiments on both simulated and real-world datasets demonstrate that our method significantly outperforms existing approaches.
  arXiv  Detail & Related papers  (2025-11-02T13:45:51Z)
• Scan, Materialize, Simulate: A Generalizable Framework for Physically Grounded Robot Planning [16.193477346643295]
  Scan, Materialize, Simulate (SMS) is a unified framework that combines 3D Gaussian Splatting for accurate scene reconstruction, visual foundation models for semantic segmentation, vision-language models for material property inference, and physics simulation for reliable prediction of action outcomes.<n>Our results highlight the potential of bridging differentiable rendering for scene reconstruction, foundation models for semantic understanding, and physics-based simulation to achieve physically grounded robot planning across diverse settings.
  arXiv  Detail & Related papers  (2025-05-20T21:55:01Z)
• GRS: Generating Robotic Simulation Tasks from Real-World Images [21.599606995763036]
  GRS creates digital twin simulations from single RGB-D observations with solvable tasks for virtual agent training.<n>We ensure simulation-task alignment through generated test suites and introduce a router that iteratively refines both simulation and test code.
  arXiv  Detail & Related papers  (2024-10-20T23:33:06Z)
• Nonprehensile Riemannian Motion Predictive Control [57.295751294224765]
  We introduce a novel Real-to-Sim reward analysis technique to reliably imagine and predict the outcome of taking possible actions for a real robotic platform. We produce a closed-loop controller to reactively push objects in a continuous action space. We observe that RMPC is robust in cluttered as well as occluded environments and outperforms the baselines.
  arXiv  Detail & Related papers  (2021-11-15T18:50:04Z)
• Auto-Tuned Sim-to-Real Transfer [143.44593793640814]
  Policies trained in simulation often fail when transferred to the real world. Current approaches to tackle this problem, such as domain randomization, require prior knowledge and engineering. We propose a method for automatically tuning simulator system parameters to match the real world.
  arXiv  Detail & Related papers  (2021-04-15T17:59:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.