Phys2Real: Fusing VLM Priors with Interactive Online Adaptation for Uncertainty-Aware Sim-to-Real Manipulation

• URL: http://arxiv.org/abs/2510.11689v1
• Date: Mon, 13 Oct 2025 17:51:23 GMT
• Title: Phys2Real: Fusing VLM Priors with Interactive Online Adaptation for Uncertainty-Aware Sim-to-Real Manipulation
• Abstract summary: Phys2Real is a real-to-sim-to-real RL pipeline that combines vision-language model (VLM)-inferred physical parameter estimates with interactive adaptation through uncertainty-aware fusion.<n>Our approach consists of three core components: (1) high-fidelity geometric reconstruction with 3D Gaussian splatting, (2) VLM-inferred prior over physical parameters, and (3) online physical parameter estimation from interaction data.
• Score: 17.786858357661604
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Learning robotic manipulation policies directly in the real world can be expensive and time-consuming. While reinforcement learning (RL) policies trained in simulation present a scalable alternative, effective sim-to-real transfer remains challenging, particularly for tasks that require precise dynamics. To address this, we propose Phys2Real, a real-to-sim-to-real RL pipeline that combines vision-language model (VLM)-inferred physical parameter estimates with interactive adaptation through uncertainty-aware fusion. Our approach consists of three core components: (1) high-fidelity geometric reconstruction with 3D Gaussian splatting, (2) VLM-inferred prior distributions over physical parameters, and (3) online physical parameter estimation from interaction data. Phys2Real conditions policies on interpretable physical parameters, refining VLM predictions with online estimates via ensemble-based uncertainty quantification. On planar pushing tasks of a T-block with varying center of mass (CoM) and a hammer with an off-center mass distribution, Phys2Real achieves substantial improvements over a domain randomization baseline: 100% vs 79% success rate for the bottom-weighted T-block, 57% vs 23% in the challenging top-weighted T-block, and 15% faster average task completion for hammer pushing. Ablation studies indicate that the combination of VLM and interaction information is essential for success. Project website: https://phys2real.github.io/ .

Related papers

• Enhancing Sim2Real Transfer for Torque-Controlled Robots through Real2Sim Dynamics Estimation and Reinforcement Learning [2.071516130824992]
  Transferring reinforcement learning policies from simulation to Real-World robots remains a major challenge.<n>We propose a Real2Sim2Real pipeline that improves Sim2Real transfer for torque-controlled robotic arms.<n>This work highlights the effectiveness of accurate physical modelling in enabling stable and generalizable torque-based reinforcement learning policies.
  arXiv  Detail & Related papers  (2026-08-23T22:25:33Z)
• Learning Physics-Guided Residual Dynamics for Deformable Object Simulation [19.104444935269616]
  Physics-Guided Residual Dynamics (PGRD) is a hybrid simulation framework that combines the advantages of physics-based and learning-based approaches.<n>We show that PGRD produces more accurate results than both purely physics-based and learning-based methods on a set of diverse real-world deformable objects.
  arXiv  Detail & Related papers  (2026-07-15T05:15:43Z)
• PLUME: Probabilistic Latent Unified World Modeling and Parameter Estimation for Multi-Finger Manipulation [15.86252454772027]
  We propose Probabilistic Latent Unified world Modeling and parameter Estimation (PLUME)<n>PLUME learns to evolve a belief over parameter values as well as the system dynamics conditioned on those parameters.<n>We evaluate our method on simulated screwdriver turning, valve turning, bucket lifting, and disk flicking tasks.
  arXiv  Detail & Related papers  (2026-06-09T19:36:25Z)
• MoSA: Motion-constrained Stress Adaptation for Mitigating Real-to-Sim Gap in Continuum Dynamics via Learning Residual Anisotropy [37.207913635032604]
  MoSA is a motion-constrained stress adaptation framework.<n>It learns residual stress operators to capture mild anisotropy and heterogeneity.<n>It achieves superior accuracy, generalization, and robustness, while learning physically meaningful residual anisotropy.
  arXiv  Detail & Related papers  (2026-05-21T15:13:49Z)
• SIM1: Physics-Aligned Simulator as Zero-Shot Data Scaler in Deformable Worlds [36.87909899472009]
  We introduce SIM1, a physics-aligned real-to-sim-to-real data engine that grounds simulation in the physical world.<n>The system digitizes scenes into metric-consistent twins, calibrates deformable dynamics through elastic modeling, and expands behaviors via diffusion-based trajectory generation.<n>Experiments show that policies trained on purely synthetic data achieve parity with real-data baselines at a 1:15 equivalence ratio, while delivering 90% zero-shot success and 50% generalization gains in real-world deployment.
  arXiv  Detail & Related papers  (2026-04-09T17:59:52Z)
• D-REX: Differentiable Real-to-Sim-to-Real Engine for Learning Dexterous Grasping [66.22412592525369]
  We introduce a real-to-sim-to-real engine that leverages the Gaussian Splat representations to build a differentiable engine.<n>We show that our engine achieves accurate and robust performance in mass identification across various object geometries and mass values.<n>Those optimized mass values facilitate force-aware policy learning, achieving superior and high performance in object grasping.
  arXiv  Detail & Related papers  (2026-03-01T15:32:04Z)
• InternVLA-A1: Unifying Understanding, Generation and Action for Robotic Manipulation [77.07565723756119]
  We present InternVLA-A1, a vision-language-action model with dynamic predictive capabilities.<n>We pre-train these models on heterogeneous data sources over real-world robot data, synthetic simulation data, and human videos.<n>We evaluate InternVLA-A1 on 12 real-world robotic tasks and a simulation benchmark.
  arXiv  Detail & Related papers  (2026-01-05T18:54:29Z)
• SIMPACT: Simulation-Enabled Action Planning using Vision-Language Models [60.80050275581661]
  Vision-Language Models (VLMs) exhibit remarkable common-sense and semantic reasoning capabilities.<n>They lack a grounded understanding of physical dynamics.<n>We present S, a test-time, SIMulation-enabled ACTion Planning framework.<n>Our method demonstrates state-of-the-art performance on five challenging, real-world rigid-body and deformable manipulation tasks.
  arXiv  Detail & Related papers  (2025-12-05T18:51:03Z)
• URDF-Anything: Constructing Articulated Objects with 3D Multimodal Language Model [76.08429266631823]
  We propose an end-to-end automatic reconstruction framework based on a 3D multimodal large language model (MLLM)<n>URDF-Anything utilizes an autoregressive prediction framework based on point-cloud and text multimodal input to jointly optimize geometric segmentation and kinematic parameter prediction.<n> Experiments on both simulated and real-world datasets demonstrate that our method significantly outperforms existing approaches.
  arXiv  Detail & Related papers  (2025-11-02T13:45:51Z)
• EmbodieDreamer: Advancing Real2Sim2Real Transfer for Policy Training via Embodied World Modeling [30.706647962741826]
  EmbodieDreamer is a novel framework that reduces the Real2Sim2Real gap from both the physics and appearance perspectives.<n>Specifically, we propose PhysAligner, a differentiable physics module designed to reduce the Real2Sim physical gap.<n>In addition, we introduce VisAligner, which incorporates a conditional video diffusion model to bridge the Sim2Real appearance gap.
  arXiv  Detail & Related papers  (2025-07-07T16:58:17Z)
• Hybrid Neural-MPM for Interactive Fluid Simulations in Real-Time [57.30651532625017]
  We present a novel hybrid method that integrates numerical simulation, neural physics, and generative control.<n>Our system demonstrates robust performance across diverse 2D/3D scenarios, material types, and obstacle interactions.<n>We promise to release both models and data upon acceptance.
  arXiv  Detail & Related papers  (2025-05-25T01:27:18Z)
• Dynamic Manipulation of Deformable Objects in 3D: Simulation, Benchmark and Learning Strategy [88.8665000676562]
  Prior methods often simplify the problem to low-speed or 2D settings, limiting their applicability to real-world 3D tasks.<n>To mitigate data scarcity, we introduce a novel simulation framework and benchmark grounded in reduced-order dynamics.<n>We propose Dynamics Informed Diffusion Policy (DIDP), a framework that integrates imitation pretraining with physics-informed test-time adaptation.
  arXiv  Detail & Related papers  (2025-05-23T03:28:25Z)
• GaussianProperty: Integrating Physical Properties to 3D Gaussians with LMMs [21.3615403516602]
  Estimating physical properties for visual data is a crucial task in computer vision, graphics, and robotics.<n>We introduce GaussianProperty, a training-free framework that assigns physical properties of materials to 3D Gaussians.<n>We demonstrate that 3D Gaussians with physical property annotations enable applications in physics-based dynamic simulation and robotic grasping.
  arXiv  Detail & Related papers  (2024-12-15T17:44:10Z)
• Physics Context Builders: A Modular Framework for Physical Reasoning in Vision-Language Models [9.474337395173388]
  Physical reasoning remains a significant challenge for Vision-Language Models (VLMs)<n>Fine-tuning is expensive for large models and impractical to repeatedly perform for every task.<n>We introduce Physics Context Builders (PCBs), a novel modular framework where specialized VLMs are fine-tuned to generate detailed physical scene descriptions.
  arXiv  Detail & Related papers  (2024-12-11T18:40:16Z)
• DeepSimHO: Stable Pose Estimation for Hand-Object Interaction via Physics Simulation [81.11585774044848]
  We present DeepSimHO, a novel deep-learning pipeline that combines forward physics simulation and backward gradient approximation with a neural network. Our method noticeably improves the stability of the estimation and achieves superior efficiency over test-time optimization.
  arXiv  Detail & Related papers  (2023-10-11T05:34:36Z)
• AdaptSim: Task-Driven Simulation Adaptation for Sim-to-Real Transfer [10.173835871228718]
  AdaptSim aims to optimize task performance in target (real) environments. First, we meta-learn an adaptation policy in simulation using reinforcement learning. We then perform iterative real-world adaptation by inferring new simulation parameter distributions for policy training.
  arXiv  Detail & Related papers  (2023-02-09T19:10:57Z)
• Learning Physical Dynamics with Subequivariant Graph Neural Networks [99.41677381754678]
  Graph Neural Networks (GNNs) have become a prevailing tool for learning physical dynamics. Physical laws abide by symmetry, which is a vital inductive bias accounting for model generalization. Our model achieves on average over 3% enhancement in contact prediction accuracy across 8 scenarios on Physion and 2X lower rollout MSE on RigidFall.
  arXiv  Detail & Related papers  (2022-10-13T10:00:30Z)
• Fast Modeling and Understanding Fluid Dynamics Systems with Encoder-Decoder Networks [0.0]
  We show that an accurate deep-learning-based proxy model can be taught efficiently by a finite-volume-based simulator. Compared to traditional simulation, the proposed deep learning approach enables much faster forward computation. We quantify the sensitivity of the deep learning model to key physical parameters and hence demonstrate that the inversion problems can be solved with great acceleration.
  arXiv  Detail & Related papers  (2020-06-09T17:14:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.