RoboSnap: One-Shot Real-to-Sim Scene Generation for Generalizable Robot Learning and Evaluation

• URL: http://arxiv.org/abs/2607.06699v1
• Date: Tue, 07 Jul 2026 18:19:10 GMT
• Title: RoboSnap: One-Shot Real-to-Sim Scene Generation for Generalizable Robot Learning and Evaluation
• Abstract summary: We present RoboSnap, a real-to-sim framework that turns a single RGB image into a simulation-ready scene.<n>Experiments on DROID scenes and real-robot tasks show that RoboSnap achieves reliable trajectory replay in the recovered scenes.<n>To further support real-to-sim research, we introduce DROID-Sim, a real-to-sim companion dataset constructed from 564 real-world scenes in DROID.
• Score: 55.07260698542592
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recovering real-world scenes as interactive simulation environments can enable generalizable robot learning and reproducible policy evaluation. However, constructing scenes that are both physically stable and visually faithful remains slow and expensive. In this work, we present RoboSnap, a real-to-sim framework that turns a single RGB image into a simulation-ready scene. The key idea is a layered design that separates the physics-critical interaction area from the surrounding visual context: collision-aware foreground assets are refined for stable robot interaction, while a 3D Gaussian splatting visual layer preserves faithful background appearance under novel views. Experiments on DROID scenes and real-robot tasks show that RoboSnap achieves reliable trajectory replay in the recovered scenes, supports task-specific synthetic data generation for policy training, and yields meaningful sim-real correlation for policy evaluation. To further support real-to-sim research, we introduce DROID-Sim, a real-to-sim companion dataset constructed from 564 real-world scenes in DROID. Extensive experiments suggest that the value of real-to-sim methods lies not only in high-fidelity visual reconstruction, but in turning real environments into reusable infrastructure for robot learning and evaluation.

Related papers

• RGB-D Video Generation for Improving Human-to-Robot Object Handover Prediction [51.76737991200891]
  Human-to-robot (H2R) object handover is a fundamental capability for human-robot collaboration, yet progress is hindered by the scarcity of large-scale, human-centric datasets.<n>We introduce Hand2Bot, an RGB-D video dataset that provides rich contextual information such as body posture and facial expressions.<n>We propose PassGen, a generative pipeline that synthesizes realistic handover sequences while ensuring hand-object consistency.
  arXiv  Detail & Related papers  (2026-08-13T09:55:59Z)
• GASE: Gaussian Splatting-Based Automated System for Reconstructing Embodied-Simulation Environments [63.222920701596564]
  Training embodied agents in the real world requires skilled operators and expensive hardware.<n> Simulation environments offer a compelling alternative by enabling large-scale, cost-effective data augmentation.<n>Fastly constructing high-fidelity simulation scenes with a minimal sim-to-real gap has become a critical objective in robot learning.
  arXiv  Detail & Related papers  (2026-06-16T05:00:42Z)
• Toward Visually Realistic Simulation: A Benchmark for Evaluating Robot Manipulation in Simulation [62.51953630639423]
  Existing benchmarks lack visual realism, creating a large domain gap between simulation and reality.<n>We propose VISER, a visually realistic benchmark for evaluating robot manipulation in simulation.<n>VISER features a high-fidelity dataset of over 1,000 3D assets with physically-based rendering (PBR) materials, along with 3D scenes created from these assets through curated layouts or generation.
  arXiv  Detail & Related papers  (2026-05-07T14:13:05Z)
• From Seeing to Simulating: Generative High-Fidelity Simulation with Digital Cousins for Generalizable Robot Learning and Evaluation [11.382656265384854]
  We present a generative framework that establishes a generative real-to-sim mapping from real-world panoramas to high-fidelity simulation scenes.<n>Combined with high-quality physics engines and realistic assets, the generated scenes support interactive manipulation tasks.<n>Experiments demonstrate a strong sim-to-real correlation validating our platform's fidelity, and show that extensively scaling up data generation leads to significantly better generalization to unseen scene and object variations.
  arXiv  Detail & Related papers  (2026-04-17T08:06:26Z)
• PolaRiS: Scalable Real-to-Sim Evaluations for Generalist Robot Policies [88.78188489161028]
  We introduce Policy Evaluation and Environment Reconstruction in Simulation (PolaRiS)<n>PolaRiS is a scalable real-to-sim framework for high-fidelity simulated robot evaluation.<n>We show that PolaRiS evaluations provide a much stronger correlation to real world generalist policy performance than existing simulated benchmarks.
  arXiv  Detail & Related papers  (2025-12-18T18:49:41Z)
• Real-to-Sim Robot Policy Evaluation with Gaussian Splatting Simulation of Soft-Body Interactions [27.247431258140463]
  We present a real-to-sim policy evaluation framework that constructs soft-body digital twins from real-world videos.<n>We validate our approach on representative deformable manipulation tasks, including plush toy packing, rope routing, and T-block pushing.
  arXiv  Detail & Related papers  (2025-11-06T18:52:08Z)
• EmbodiedSplat: Personalized Real-to-Sim-to-Real Navigation with Gaussian Splats from a Mobile Device [33.22697339175522]
  Embodied AI predominantly relies on simulation for training and evaluation.<n>Sim-to-real transfer remains a major challenge.<n>EmbodiedSplat is a novel approach that personalizes policy training.
  arXiv  Detail & Related papers  (2025-09-22T07:22:31Z)
• Is Single-View Mesh Reconstruction Ready for Robotics? [78.14584238127338]
  We evaluate single-view mesh reconstruction models for their potential in enabling instant digital twin creation for real-time planning and dynamics prediction using physics simulators for robotic manipulation.<n>Our findings highlight critical gaps between computer vision advances and robotics needs, guiding future research at this intersection.
  arXiv  Detail & Related papers  (2025-05-23T14:35:56Z)
• VR-Robo: A Real-to-Sim-to-Real Framework for Visual Robot Navigation and Locomotion [25.440573256776133]
  This paper presents a Real-to-Sim-to-Real framework that generates and physically interactive "digital twin" simulation environments for visual navigation and locomotion learning.
  arXiv  Detail & Related papers  (2025-02-03T17:15:05Z)
• Learning Sim-to-Real Dense Object Descriptors for Robotic Manipulation [4.7246285569677315]
  We present Sim-to-Real Dense Object Nets (SRDONs), a dense object descriptor that not only understands the object via appropriate representation but also maps simulated and real data to a unified feature space with pixel consistency. We demonstrate in experiments that pre-trained SRDONs significantly improve performances on unseen objects and unseen visual environments for various robotic tasks with zero real-world training.
  arXiv  Detail & Related papers  (2023-04-18T02:28:55Z)
• Point Cloud Based Reinforcement Learning for Sim-to-Real and Partial Observability in Visual Navigation [62.22058066456076]
  Reinforcement Learning (RL) represents powerful tools to solve complex robotic tasks. RL does not work directly in the real-world, which is known as the sim-to-real transfer problem. We propose a method that learns on an observation space constructed by point clouds and environment randomization.
  arXiv  Detail & Related papers  (2020-07-27T17:46:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.