Synthetic vs. Real Training Data for Visual Navigation

• URL: http://arxiv.org/abs/2509.11791v1
• Date: Mon, 15 Sep 2025 11:22:40 GMT
• Title: Synthetic vs. Real Training Data for Visual Navigation
• Authors: Lauri Suomela, Sasanka Kuruppu Arachchige, German F. Torres, Harry Edelman, Joni-Kristian Kämäräinen,
• Abstract summary: This paper investigates how the performance of visual navigation policies trained in simulation compares to policies trained with real-world data.<n>We use a navigation policy architecture that bridges the sim-to-real appearance gap by leveraging pretrained visual representations and runs real-time on robot hardware.<n>Our results highlight the importance of diverse image encoder pretraining for sim-to-real generalization, and identify on-policy learning as a key advantage of simulated training over training with real data.
• Score: 6.5298097830674635
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper investigates how the performance of visual navigation policies trained in simulation compares to policies trained with real-world data. Performance degradation of simulator-trained policies is often significant when they are evaluated in the real world. However, despite this well-known sim-to-real gap, we demonstrate that simulator-trained policies can match the performance of their real-world-trained counterparts. Central to our approach is a navigation policy architecture that bridges the sim-to-real appearance gap by leveraging pretrained visual representations and runs real-time on robot hardware. Evaluations on a wheeled mobile robot show that the proposed policy, when trained in simulation, outperforms its real-world-trained version by 31% and the prior state-of-the-art methods by 50% in navigation success rate. Policy generalization is verified by deploying the same model onboard a drone. Our results highlight the importance of diverse image encoder pretraining for sim-to-real generalization, and identify on-policy learning as a key advantage of simulated training over training with real data.

Related papers

• Opening the Sim-to-Real Door for Humanoid Pixel-to-Action Policy Transfer [59.02729900344616]
  GPU-accelerated, photorealistic simulation has opened a scalable data-generation path for robot learning.<n>We develop a teacher-student-bootstrap learning framework for vision-based humanoid loco-manipulation.<n>This represents the first humanoid sim-to-real policy capable of diverse articulated loco-manipulation using pure RGB perception.
  arXiv  Detail & Related papers  (2025-11-30T20:07:13Z)
• Real-to-Sim Robot Policy Evaluation with Gaussian Splatting Simulation of Soft-Body Interactions [27.247431258140463]
  We present a real-to-sim policy evaluation framework that constructs soft-body digital twins from real-world videos.<n>We validate our approach on representative deformable manipulation tasks, including plush toy packing, rope routing, and T-block pushing.
  arXiv  Detail & Related papers  (2025-11-06T18:52:08Z)
• EmbodiedSplat: Personalized Real-to-Sim-to-Real Navigation with Gaussian Splats from a Mobile Device [33.22697339175522]
  Embodied AI predominantly relies on simulation for training and evaluation.<n>Sim-to-real transfer remains a major challenge.<n>EmbodiedSplat is a novel approach that personalizes policy training.
  arXiv  Detail & Related papers  (2025-09-22T07:22:31Z)
• Real-is-Sim: Bridging the Sim-to-Real Gap with a Dynamic Digital Twin [8.498460043101499]
  We introduce real-is-sim, a new approach to integrating simulation into behavior cloning pipelines.<n>In contrast to real-only methods, which lack the ability to safely test policies before deployment, and sim-to-real methods, which require complex adaptation to cross the sim-to-real gap.<n>Our framework allows policies to seamlessly switch between running on real hardware and running in parallelized virtual environments.
  arXiv  Detail & Related papers  (2025-04-04T17:05:56Z)
• Empirical Analysis of Sim-and-Real Cotraining of Diffusion Policies for Planar Pushing from Pixels [18.858676073824515]
  Cotraining with demonstration data generated both in simulation and on real hardware has emerged as a promising recipe for scaling imitation learning in robotics.<n>This work seeks to elucidate basic principles of this sim-and-real cotraining to inform simulation design, sim-and-real dataset creation, and policy training.
  arXiv  Detail & Related papers  (2025-03-28T17:25:57Z)
• Gaussian Splatting to Real World Flight Navigation Transfer with Liquid Networks [93.38375271826202]
  We present a method to improve generalization and robustness to distribution shifts in sim-to-real visual quadrotor navigation tasks. We first build a simulator by integrating Gaussian splatting with quadrotor flight dynamics, and then, train robust navigation policies using Liquid neural networks. In this way, we obtain a full-stack imitation learning protocol that combines advances in 3D Gaussian splatting radiance field rendering, programming of expert demonstration training data, and the task understanding capabilities of Liquid networks.
  arXiv  Detail & Related papers  (2024-06-21T13:48:37Z)
• Robust Visual Sim-to-Real Transfer for Robotic Manipulation [79.66851068682779]
  Learning visuomotor policies in simulation is much safer and cheaper than in the real world. However, due to discrepancies between the simulated and real data, simulator-trained policies often fail when transferred to real robots. One common approach to bridge the visual sim-to-real domain gap is domain randomization (DR)
  arXiv  Detail & Related papers  (2023-07-28T05:47:24Z)
• DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Reality [64.51295032956118]
  We train a policy that can perform robust dexterous manipulation on an anthropomorphic robot hand. Our work reaffirms the possibilities of sim-to-real transfer for dexterous manipulation in diverse kinds of hardware and simulator setups.
  arXiv  Detail & Related papers  (2022-10-25T01:51:36Z)
• Cloud-Edge Training Architecture for Sim-to-Real Deep Reinforcement Learning [0.8399688944263843]
  Deep reinforcement learning (DRL) is a promising approach to solve complex control tasks by learning policies through interactions with the environment. Sim-to-real approaches leverage simulations to pretrain DRL policies and then deploy them in the real world. This work proposes a distributed cloud-edge architecture to train DRL agents in the real world in real-time.
  arXiv  Detail & Related papers  (2022-03-04T10:27:01Z)
• Practical Imitation Learning in the Real World via Task Consistency Loss [18.827979446629296]
  This paper introduces a self-supervised loss that encourages sim and real alignment both at the feature and action-prediction levels. We achieve 80% success across ten seen and unseen scenes using only 16.2 hours of teleoperated demonstrations in sim and real.
  arXiv  Detail & Related papers  (2022-02-03T21:43:06Z)
• Point Cloud Based Reinforcement Learning for Sim-to-Real and Partial Observability in Visual Navigation [62.22058066456076]
  Reinforcement Learning (RL) represents powerful tools to solve complex robotic tasks. RL does not work directly in the real-world, which is known as the sim-to-real transfer problem. We propose a method that learns on an observation space constructed by point clouds and environment randomization.
  arXiv  Detail & Related papers  (2020-07-27T17:46:59Z)
• SimAug: Learning Robust Representations from Simulation for Trajectory Prediction [78.91518036949918]
  We propose a novel approach to learn robust representation through augmenting the simulation training data. We show that SimAug achieves promising results on three real-world benchmarks using zero real training data.
  arXiv  Detail & Related papers  (2020-04-04T21:22:01Z)
• Simulation-based reinforcement learning for real-world autonomous driving [9.773015744446067]
  We use reinforcement learning in simulation to obtain a driving system controlling a full-size real-world vehicle. The driving policy takes RGB images from a single camera and their semantic segmentation as input. We use mostly synthetic data, with labelled real-world data appearing only in the training of the segmentation network.
  arXiv  Detail & Related papers  (2019-11-29T00:08:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.