Bridging Simulation and Reality: Cross-Domain Transfer with Semantic 2D Gaussian Splatting

• URL: http://arxiv.org/abs/2512.04731v1
• Date: Thu, 04 Dec 2025 12:16:15 GMT
• Title: Bridging Simulation and Reality: Cross-Domain Transfer with Semantic 2D Gaussian Splatting
• Abstract summary: Cross-domain transfer in robotic manipulation is a longstanding challenge due to the significant domain gap between simulated and real-world environments.<n>We propose Semantic 2D Gaussian Splatting (S2GS), a novel representation method that extracts object-centric, domain-invariant spatial features.<n>S2GS significantly improves sim-to-real transferability, maintaining high and stable task performance in real-world scenarios.
• Score: 60.513168855890974
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Cross-domain transfer in robotic manipulation remains a longstanding challenge due to the significant domain gap between simulated and real-world environments. Existing methods such as domain randomization, adaptation, and sim-real calibration often require extensive tuning or fail to generalize to unseen scenarios. To address this issue, we observe that if domain-invariant features are utilized during policy training in simulation, and the same features can be extracted and provided as the input to policy during real-world deployment, the domain gap can be effectively bridged, leading to significantly improved policy generalization. Accordingly, we propose Semantic 2D Gaussian Splatting (S2GS), a novel representation method that extracts object-centric, domain-invariant spatial features. S2GS constructs multi-view 2D semantic fields and projects them into a unified 3D space via feature-level Gaussian splatting. A semantic filtering mechanism removes irrelevant background content, ensuring clean and consistent inputs for policy learning. To evaluate the effectiveness of S2GS, we adopt Diffusion Policy as the downstream learning algorithm and conduct experiments in the ManiSkill simulation environment, followed by real-world deployment. Results demonstrate that S2GS significantly improves sim-to-real transferability, maintaining high and stable task performance in real-world scenarios.

Related papers

• BIFROST: Bridging Invariant Feature Representation for Observation-space Sim2Real Transfer [12.497312159934351]
  We study whether we can identify and exploit this shared structure from raw observations to train a policy that enables zero shot transfer.<n>We introduce BIFROST, which learns a shared history encoder on paired cross-domain data via cross-domain bi objective.<n>We provide empirical evidence on sim2sim visual navigation and sim2real contact rich manipulation task and visual servoing task.
  arXiv  Detail & Related papers  (2026-07-01T19:15:17Z)
• R2RGEN: Real-to-Real 3D Data Generation for Spatially Generalized Manipulation [74.41728218960465]
  We propose a real-to-real 3D data generation framework (R2RGen) that directly augments the pointcloud observation-action pairs to generate real-world data.<n>R2RGen substantially enhances data efficiency on extensive experiments and demonstrates strong potential for scaling and application on mobile manipulation.
  arXiv  Detail & Related papers  (2025-10-09T17:55:44Z)
• Topology-Aware Modeling for Unsupervised Simulation-to-Reality Point Cloud Recognition [63.55828203989405]
  We introduce a novel Topology-Aware Modeling (TAM) framework for Sim2Real UDA on object point clouds.<n>Our approach mitigates the domain gap by leveraging global spatial topology, characterized by low-level, high-frequency 3D structures.<n>We propose an advanced self-training strategy that combines cross-domain contrastive learning with self-training.
  arXiv  Detail & Related papers  (2025-06-26T11:53:59Z)
• Bridging the Sim2Real Gap: Vision Encoder Pre-Training for Visuomotor Policy Transfer [0.0]
  We evaluate the performance of pre-trained vision encoders to address the Sim2Real gap.<n>We show that manipulation-pretrained encoders consistently achieve higher Action Scores.
  arXiv  Detail & Related papers  (2025-01-26T00:27:04Z)
• DeSiRe-GS: 4D Street Gaussians for Static-Dynamic Decomposition and Surface Reconstruction for Urban Driving Scenes [71.61083731844282]
  We present DeSiRe-GS, a self-supervised gaussian splatting representation.<n>It enables effective static-dynamic decomposition and high-fidelity surface reconstruction in complex driving scenarios.
  arXiv  Detail & Related papers  (2024-11-18T05:49:16Z)
• CTS: Sim-to-Real Unsupervised Domain Adaptation on 3D Detection [16.96201890965781]
  This paper presents a novel framework to transfer models from labeled simulation to unlabeled reality domains. Experimental results demonstrate that our proposed approach significantly enhances the sim-to-real domain adaptation capability of 3D object detection models.
  arXiv  Detail & Related papers  (2024-06-26T07:31:16Z)
• Sim-to-Real Transfer of Deep Reinforcement Learning Agents for Online Coverage Path Planning [22.077058792635313]
  Coverage path planning is the problem of finding a path that covers the entire free space of a confined area.<n>We investigate the suitability of continuous-space reinforcement learning for this challenging problem.<n>We show that our approach surpasses the performance of both previous RL-based approaches and highly specialized methods.
  arXiv  Detail & Related papers  (2024-06-07T13:24:19Z)
• Bridging the Sim2Real gap with CARE: Supervised Detection Adaptation with Conditional Alignment and Reweighting [72.75792823726479]
  We propose Conditional Domain Translation via Conditional Alignment and Reweighting (CARE) to close the sim2real appearance and content gaps. We present an analytical justification of our algorithm and demonstrate strong gains over competing methods on standard benchmarks.
  arXiv  Detail & Related papers  (2023-02-09T18:39:28Z)
• One-Shot Domain Adaptive and Generalizable Semantic Segmentation with Class-Aware Cross-Domain Transformers [96.51828911883456]
  Unsupervised sim-to-real domain adaptation (UDA) for semantic segmentation aims to improve the real-world test performance of a model trained on simulated data. Traditional UDA often assumes that there are abundant unlabeled real-world data samples available during training for the adaptation. We explore the one-shot unsupervised sim-to-real domain adaptation (OSUDA) and generalization problem, where only one real-world data sample is available.
  arXiv  Detail & Related papers  (2022-12-14T15:54:15Z)
• Data-efficient Domain Randomization with Bayesian Optimization [34.854609756970305]
  When learning policies for robot control, the required real-world data is typically prohibitively expensive to acquire. BayRn is a black-box sim-to-real algorithm that solves tasks efficiently by adapting the domain parameter distribution. Our results show that BayRn is able to perform sim-to-real transfer, while significantly reducing the required prior knowledge.
  arXiv  Detail & Related papers  (2020-03-05T07:48:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.