BIFROST: Bridging Invariant Feature Representation for Observation-space Sim2Real Transfer

• URL: http://arxiv.org/abs/2607.01410v1
• Date: Wed, 01 Jul 2026 19:15:17 GMT
• Title: BIFROST: Bridging Invariant Feature Representation for Observation-space Sim2Real Transfer
• Abstract summary: We study whether we can identify and exploit this shared structure from raw observations to train a policy that enables zero shot transfer.<n>We introduce BIFROST, which learns a shared history encoder on paired cross-domain data via cross-domain bi objective.<n>We provide empirical evidence on sim2sim visual navigation and sim2real contact rich manipulation task and visual servoing task.
• Score: 12.497312159934351
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Sim2real transfer for robot policy learning suffers due to mismatch between simulation and reality. Existing methods typically address each gap in isolation through separate adaptation modules, which are composed or layered when both gaps coexist. Yet the basis for attempting sim2real in the first place is that there is shared structure between a task in simulation and reality, where equivalent actions from equivalent configurations produce equivalent long term outcomes regardless of domain specific differences in rendering or physics. In this paper, we study whether we can identify and exploit this shared structure from raw observations to train a policy that enables zero shot transfer. We introduce BIFROST, which learns a shared history encoder on paired cross-domain data via cross-domain bisimulation objective: observation-action sequences leading to equivalent long-term behavior are mapped to nearby latent states, regardless of domain. Policies trained on these latent states in simulation transfer zero-shot to reality. We provide empirical evidence on sim2sim visual navigation and sim2real contact rich manipulation task and visual servoing task that BIFROST achieves effective transfer where domain adaptation and co-training baselines fail under both visual and dynamics domain gaps.

Related papers

• Bridging Simulation and Reality: Cross-Domain Transfer with Semantic 2D Gaussian Splatting [60.513168855890974]
  Cross-domain transfer in robotic manipulation is a longstanding challenge due to the significant domain gap between simulated and real-world environments.<n>We propose Semantic 2D Gaussian Splatting (S2GS), a novel representation method that extracts object-centric, domain-invariant spatial features.<n>S2GS significantly improves sim-to-real transferability, maintaining high and stable task performance in real-world scenarios.
  arXiv  Detail & Related papers  (2025-12-04T12:16:15Z)
• Generalizable Domain Adaptation for Sim-and-Real Policy Co-Training [21.855770200309674]
  We propose a unified sim-and-real co-training framework for learning generalizable manipulation policies.<n>We show it can leverage abundant simulation data to achieve up to a 30% improvement in the real-world success rate.
  arXiv  Detail & Related papers  (2025-09-23T04:32:53Z)
• Topology-Aware Modeling for Unsupervised Simulation-to-Reality Point Cloud Recognition [63.55828203989405]
  We introduce a novel Topology-Aware Modeling (TAM) framework for Sim2Real UDA on object point clouds.<n>Our approach mitigates the domain gap by leveraging global spatial topology, characterized by low-level, high-frequency 3D structures.<n>We propose an advanced self-training strategy that combines cross-domain contrastive learning with self-training.
  arXiv  Detail & Related papers  (2025-06-26T11:53:59Z)
• xTED: Cross-Domain Adaptation via Diffusion-Based Trajectory Editing [21.37585797507323]
  Cross-domain policy transfer methods mostly aim at learning domain correspondences or corrections to facilitate policy learning.<n>We propose the Cross-Domain Trajectory EDiting framework that employs a specially designed diffusion model for cross-domain trajectory adaptation.<n>Our proposed model architecture effectively captures the intricate dependencies among states, actions, and rewards, as well as the dynamics patterns within target data.
  arXiv  Detail & Related papers  (2024-09-13T10:07:28Z)
• One-Shot Domain Adaptive and Generalizable Semantic Segmentation with Class-Aware Cross-Domain Transformers [96.51828911883456]
  Unsupervised sim-to-real domain adaptation (UDA) for semantic segmentation aims to improve the real-world test performance of a model trained on simulated data. Traditional UDA often assumes that there are abundant unlabeled real-world data samples available during training for the adaptation. We explore the one-shot unsupervised sim-to-real domain adaptation (OSUDA) and generalization problem, where only one real-world data sample is available.
  arXiv  Detail & Related papers  (2022-12-14T15:54:15Z)
• Provable Sim-to-real Transfer in Continuous Domain with Partial Observations [39.18274543757048]
  Sim-to-real transfer trains RL agents in the simulated environments and then deploys them in the real world. We show that a popular robust adversarial training algorithm is capable of learning a policy from the simulated environment that is competitive to the optimal policy in the real-world environment.
  arXiv  Detail & Related papers  (2022-10-27T16:37:52Z)
• Cross-domain Imitation from Observations [50.669343548588294]
  Imitation learning seeks to circumvent the difficulty in designing proper reward functions for training agents by utilizing expert behavior. In this paper, we study the problem of how to imitate tasks when there exist discrepancies between the expert and agent MDP. We present a novel framework to learn correspondences across such domains.
  arXiv  Detail & Related papers  (2021-05-20T21:08:25Z)
• Learning Cross-Domain Correspondence for Control with Dynamics Cycle-Consistency [60.39133304370604]
  We learn to align dynamic robot behavior across two domains using a cycle-consistency constraint. Our framework is able to align uncalibrated monocular video of a real robot arm to dynamic state-action trajectories of a simulated arm without paired data.
  arXiv  Detail & Related papers  (2020-12-17T18:22:25Z)
• Sim2Real for Self-Supervised Monocular Depth and Segmentation [7.376636976924]
  Image-based learning methods for autonomous vehicle perception tasks require large quantities of labelled, real data in order to properly train without overfitting. Recent advances in domain adaptation have indicated that a shared latent space assumption can help to bridge the gap between the simulation and real domains. We demonstrate that a twin VAE-based architecture with a shared latent space and auxiliary decoders is able to bridge the sim2real gap without requiring any paired, ground-truth data in the real domain.
  arXiv  Detail & Related papers  (2020-12-01T03:25:02Z)
• From Simulation to Real World Maneuver Execution using Deep Reinforcement Learning [69.23334811890919]
  Deep Reinforcement Learning has proved to be able to solve many control tasks in different fields, but the behavior of these systems is not always as expected when deployed in real-world scenarios. This is mainly due to the lack of domain adaptation between simulated and real-world data together with the absence of distinction between train and test datasets. We present a system based on multiple environments in which agents are trained simultaneously, evaluating the behavior of the model in different scenarios.
  arXiv  Detail & Related papers  (2020-05-13T14:22:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.