Efficient Sim-to-Real Transfer of World-Action Models from Synthetic Priors

• URL: http://arxiv.org/abs/2606.31101v1
• Date: Tue, 30 Jun 2026 03:49:31 GMT
• Title: Efficient Sim-to-Real Transfer of World-Action Models from Synthetic Priors
• Abstract summary: We study whether a world-action model can be trained from synthetic priors and deployed zero-shot in the real world.<n>We construct simulation environments with extensive domain randomization and generate demonstrations using the AnyTask motion planning pipeline.<n>When deployed zero-shot on a Franka Robot, our policy attains a 35% average success rate.
• Score: 15.482859998546985
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Bridging the sim-to-real gap is a core challenge in deploying learned manipulation policies. Sim-to-real learning is attractive because it can replace expensive real robot demonstrations with scalable synthetic data, yet world-action models have not previously been shown to transfer from simulation to real robotic manipulation. We study whether a world-action model can be trained from synthetic priors and deployed zero-shot in the real world. To this end, we build upon Cosmos Policy, a video diffusion model adapted for visuomotor control. We construct simulation environments with extensive domain randomization and generate demonstrations using the AnyTask motion planning pipeline. We evaluate our approach across object lifting, drawer opening, and pick-and-place tasks using ${\sim}800$ synthetic demonstrations per task and no real demonstrations. When deployed zero-shot on a Franka Robot, our policy attains a 35\% average success rate. To our knowledge, this represents the first successful sim-to-real transfer of a world-action model for robotic manipulation.

Related papers

• OASIS: From Simulation Data Collection to Real-World Humanoid Loco-Manipulation [51.37436817948304]
  OASIS is a simulation-data-driven framework for humanoid loco-manipulation.<n>It reconstructs realistic object assets from real-world images using a 3D generative model.<n>Under zero-shot deployment, it achieves higher success rates on most tasks than that trained on real-robot teleoperation data.
  arXiv  Detail & Related papers  (2026-06-07T10:01:53Z)
• ComSim: Building Scalable Real-World Robot Data Generation via Compositional Simulation [90.4702774169675]
  We propose a novel hybrid approach called Compositional Simulation, which combines classical simulation and neural simulation.<n>Our approach utilizes a closed-loop real-sim-real data augmentation pipeline, leveraging a small amount of real-world data.<n>We train a neural simulator to transform classical simulation videos into real-world representations, improving the accuracy of policy models trained in real-world environments.
  arXiv  Detail & Related papers  (2026-04-13T12:25:45Z)
• Generative Simulation for Policy Learning in Physical Human-Robot Interaction [11.978357519356537]
  We introduce a zero-shot "text2sim2real" generative simulation framework that automatically synthesizes diverse pHRI scenarios.<n>We utilize this framework to autonomously collect large-scale synthetic demonstration datasets.<n>Our learned policies successfully achieve zero-shot sim-to-real transfer, attaining success rates exceeding 80%.
  arXiv  Detail & Related papers  (2026-04-09T18:00:05Z)
• RoboPaint: From Human Demonstration to Any Robot and Any View [9.083647729839688]
  We propose a Real-Sim-Real data collection and data editing pipeline to transform human demonstrations into robot-executable, environment-specific training data.<n>We show that retargeted dex-hand trajectories achieve an 84% success rate across 10 diverse object manipulation tasks.<n>We offer a scalable, cost-effective alternative to teleoperation with minimal performance loss for complex dexterous manipulation.
  arXiv  Detail & Related papers  (2026-02-05T05:45:12Z)
• Sim-and-Human Co-training for Data-Efficient and Generalizable Robotic Manipulation [113.13282853889818]
  SimHum is a framework to simultaneously extract kinematic prior from simulated robot actions and visual prior from real-world human observations.<n>Based on the two complementary priors, we achieve data-efficient and generalizable robotic manipulation in real-world tasks.
  arXiv  Detail & Related papers  (2026-01-27T09:41:28Z)
• DreamGen: Unlocking Generalization in Robot Learning through Video World Models [120.25799361925387]
  DreamGen is a pipeline for training robot policies that generalize across behaviors and environments through neural trajectories.<n>Our work establishes a promising new axis for scaling robot learning well beyond manual data collection.
  arXiv  Detail & Related papers  (2025-05-19T04:55:39Z)
• Dream to Manipulate: Compositional World Models Empowering Robot Imitation Learning with Imagination [25.62602420895531]
  DreMa is a new approach for constructing digital twins using learned explicit representations of the real world and its dynamics.<n>We show that DreMa can successfully learn novel physical tasks from just a single example per task variation.
  arXiv  Detail & Related papers  (2024-12-19T15:38:15Z)
• DemoStart: Demonstration-led auto-curriculum applied to sim-to-real with multi-fingered robots [15.034811470942962]
  We present DemoStart, a novel auto-curriculum reinforcement learning method capable of learning complex manipulation behaviors on an arm equipped with a three-fingered robotic hand. Learning from simulation drastically reduces the development cycle of behavior generation, and domain randomization techniques are leveraged to achieve successful zero-shot sim-to-real transfer.
  arXiv  Detail & Related papers  (2024-09-10T16:05:25Z)
• DrEureka: Language Model Guided Sim-To-Real Transfer [64.14314476811806]
  Transferring policies learned in simulation to the real world is a promising strategy for acquiring robot skills at scale. In this paper, we investigate using Large Language Models (LLMs) to automate and accelerate sim-to-real design. Our approach is capable of solving novel robot tasks, such as quadruped balancing and walking atop a yoga ball.
  arXiv  Detail & Related papers  (2024-06-04T04:53:05Z)
• TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correction [25.36756787147331]
  Learning in simulation and transferring the learned policy to the real world has the potential to enable generalist robots. We propose a data-driven approach to enable successful sim-to-real transfer based on a human-in-the-loop framework. We show that our approach can achieve successful sim-to-real transfer in complex and contact-rich manipulation tasks such as furniture assembly.
  arXiv  Detail & Related papers  (2024-05-16T17:59:07Z)
• Reactive Long Horizon Task Execution via Visual Skill and Precondition Models [59.76233967614774]
  We describe an approach for sim-to-real training that can accomplish unseen robotic tasks using models learned in simulation to ground components of a simple task planner. We show an increase in success rate from 91.6% to 98% in simulation and from 10% to 80% success rate in the real-world as compared with naive baselines.
  arXiv  Detail & Related papers  (2020-11-17T15:24:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.