GSWorld: Closed-Loop Photo-Realistic Simulation Suite for Robotic Manipulation

• URL: http://arxiv.org/abs/2510.20813v1
• Date: Thu, 23 Oct 2025 17:59:26 GMT
• Title: GSWorld: Closed-Loop Photo-Realistic Simulation Suite for Robotic Manipulation
• Authors: Guangqi Jiang, Haoran Chang, Ri-Zhao Qiu, Yutong Liang, Mazeyu Ji, Jiyue Zhu, Zhao Dong, Xueyan Zou, Xiaolong Wang,
• Abstract summary: GSWorld is a photo-realistic simulator for robotics manipulation that combines 3D Gaussian Splatting with physics engines.<n>Our framework advocates "closing the loop" of developing manipulation policies with reproducible evaluation of policies learned from real-robot data.
• Score: 18.684526752120412
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper presents GSWorld, a robust, photo-realistic simulator for robotics manipulation that combines 3D Gaussian Splatting with physics engines. Our framework advocates "closing the loop" of developing manipulation policies with reproducible evaluation of policies learned from real-robot data and sim2real policy training without using real robots. To enable photo-realistic rendering of diverse scenes, we propose a new asset format, which we term GSDF (Gaussian Scene Description File), that infuses Gaussian-on-Mesh representation with robot URDF and other objects. With a streamlined reconstruction pipeline, we curate a database of GSDF that contains 3 robot embodiments for single-arm and bimanual manipulation, as well as more than 40 objects. Combining GSDF with physics engines, we demonstrate several immediate interesting applications: (1) learning zero-shot sim2real pixel-to-action manipulation policy with photo-realistic rendering, (2) automated high-quality DAgger data collection for adapting policies to deployment environments, (3) reproducible benchmarking of real-robot manipulation policies in simulation, (4) simulation data collection by virtual teleoperation, and (5) zero-shot sim2real visual reinforcement learning. Website: https://3dgsworld.github.io/.

Related papers

• Real-to-Sim Robot Policy Evaluation with Gaussian Splatting Simulation of Soft-Body Interactions [27.247431258140463]
  We present a real-to-sim policy evaluation framework that constructs soft-body digital twins from real-world videos.<n>We validate our approach on representative deformable manipulation tasks, including plush toy packing, rope routing, and T-block pushing.
  arXiv  Detail & Related papers  (2025-11-06T18:52:08Z)
• Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation [87.91642226587294]
  Current learning-based 3D reconstruction methods rely on the availability of captured real-world multi-view data.<n>We propose a self-distillation framework that distills the implicit 3D knowledge in the video diffusion models into an explicit 3D Gaussian Splatting (3DGS) representation.<n>Our framework achieves state-of-the-art performance in static and dynamic 3D scene generation.
  arXiv  Detail & Related papers  (2025-09-23T17:58:01Z)
• GWM: Towards Scalable Gaussian World Models for Robotic Manipulation [53.51622803589185]
  We propose a novel branch of world model named Gaussian World Model (GWM) for robotic manipulation.<n>At its core is a latent Diffusion Transformer (DiT) combined with a 3D variational autoencoder, enabling fine-grained scene-level future state reconstruction.<n>Both simulated and real-world experiments depict that GWM can precisely predict future scenes conditioned on diverse robot actions.
  arXiv  Detail & Related papers  (2025-08-25T02:01:09Z)
• Learning human-to-robot handovers through 3D scene reconstruction [28.930178662944446]
  We propose the first method for learning supervised-based robot handovers solely from RGB images.<n>We generate robot demonstrations containing image-action pairs captured with a camera mounted on the robot gripper.<n>As a result, the simulated camera pose changes in the reconstructed scene can be directly translated into gripper pose changes.
  arXiv  Detail & Related papers  (2025-07-11T16:26:31Z)
• RoboPearls: Editable Video Simulation for Robot Manipulation [81.18434338506621]
  RoboPearls is an editable video simulation framework for robotic manipulation.<n>Built on 3D Gaussian Splatting (3DGS), RoboPearls enables the construction of photo-realistic, view-consistent simulations.<n>We conduct extensive experiments on multiple datasets and scenes, including RLBench, COLOSSEUM, Ego4D, Open X-Embodiment, and a real-world robot.
  arXiv  Detail & Related papers  (2025-06-28T05:03:31Z)
• X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Real [20.561250366126625]
  X-Sim is a real-to-sim-to-real framework that uses object motion as a dense and transferable signal for learning robot policies.<n>X-Sim starts by reconstructing a simulation from an RGBD human video and tracking object trajectories to define object-centric rewards.<n>The learned policy is then distilled into an image-conditioned diffusion policy using synthetic rollouts rendered with varied viewpoints and lighting.
  arXiv  Detail & Related papers  (2025-05-11T19:04:00Z)
• Video2Policy: Scaling up Manipulation Tasks in Simulation through Internet Videos [61.925837909969815]
  We introduce Video2Policy, a novel framework that leverages internet RGB videos to reconstruct tasks based on everyday human behavior.<n>Our method can successfully train RL policies on such tasks, including complex and challenging tasks such as throwing.<n>We show that the generated simulation data can be scaled up for training a general policy, and it can be transferred back to the real robot in a Real2Sim2Real way.
  arXiv  Detail & Related papers  (2025-02-14T03:22:03Z)
• RoboGSim: A Real2Sim2Real Robotic Gaussian Splatting Simulator [27.04267700576422]
  RoboGSim is a real2sim2real robotic simulator powered by 3D Gaussian Splatting and the physics engine.<n>It can synthesize the simulated data with novel views, objects, trajectories, and scenes.<n>We compared the test results of RoboGSim data and real robot data on both RoboGSim and real robot platforms.
  arXiv  Detail & Related papers  (2024-11-18T18:58:03Z)
• Robust Visual Sim-to-Real Transfer for Robotic Manipulation [79.66851068682779]
  Learning visuomotor policies in simulation is much safer and cheaper than in the real world. However, due to discrepancies between the simulated and real data, simulator-trained policies often fail when transferred to real robots. One common approach to bridge the visual sim-to-real domain gap is domain randomization (DR)
  arXiv  Detail & Related papers  (2023-07-28T05:47:24Z)
• DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Reality [64.51295032956118]
  We train a policy that can perform robust dexterous manipulation on an anthropomorphic robot hand. Our work reaffirms the possibilities of sim-to-real transfer for dexterous manipulation in diverse kinds of hardware and simulator setups.
  arXiv  Detail & Related papers  (2022-10-25T01:51:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.