Understanding Domain Randomization for Sim-to-real Transfer
- URL: http://arxiv.org/abs/2110.03239v1
- Date: Thu, 7 Oct 2021 07:45:59 GMT
- Title: Understanding Domain Randomization for Sim-to-real Transfer
- Authors: Xiaoyu Chen, Jiachen Hu, Chi Jin, Lihong Li, Liwei Wang
- Abstract summary: We propose a theoretical framework for sim-to-real transfers, in which the simulator is modeled as a set of MDPs with tunable parameters.
We prove that sim-to-real transfer can succeed under mild conditions without any real-world training samples.
- Score: 41.33483293243257
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Reinforcement learning encounters many challenges when applied directly in
the real world. Sim-to-real transfer is widely used to transfer the knowledge
learned from simulation to the real world. Domain randomization -- one of the
most popular algorithms for sim-to-real transfer -- has been demonstrated to be
effective in various tasks in robotics and autonomous driving. Despite its
empirical successes, theoretical understanding on why this simple algorithm
works is limited. In this paper, we propose a theoretical framework for
sim-to-real transfers, in which the simulator is modeled as a set of MDPs with
tunable parameters (corresponding to unknown physical parameters such as
friction). We provide sharp bounds on the sim-to-real gap -- the difference
between the value of policy returned by domain randomization and the value of
an optimal policy for the real world. We prove that sim-to-real transfer can
succeed under mild conditions without any real-world training samples. Our
theory also highlights the importance of using memory (i.e., history-dependent
policies) in domain randomization. Our proof is based on novel techniques that
reduce the problem of bounding the sim-to-real gap to the problem of designing
efficient learning algorithms for infinite-horizon MDPs, which we believe are
of independent interest.
Related papers
- Overcoming the Sim-to-Real Gap: Leveraging Simulation to Learn to Explore for Real-World RL [25.991354823569033]
We show that in many regimes, while direct sim2real transfer may fail, we can utilize the simulator to learn a set of emphexploratory policies.
In particular, in the setting of low-rank MDPs, we show that coupling these exploratory policies with simple, practical approaches.
This is the first evidence that simulation transfer yields a provable gain in reinforcement learning in settings where direct sim2real transfer fails.
arXiv Detail & Related papers (2024-10-26T19:12:27Z) - Sim-to-Real Transfer of Deep Reinforcement Learning Agents for Online Coverage Path Planning [15.792914346054502]
We tackle the challenge of sim-to-real transfer of reinforcement learning (RL) agents for coverage path planning ( CPP)
We bridge the sim-to-real gap through a semi-virtual environment, including a real robot and real-time aspects, while utilizing a simulated sensor and obstacles.
We find that a high inference frequency allows first-order Markovian policies to transfer directly from simulation, while higher-order policies can be fine-tuned to further reduce the sim-to-real gap.
arXiv Detail & Related papers (2024-06-07T13:24:19Z) - DrEureka: Language Model Guided Sim-To-Real Transfer [64.14314476811806]
Transferring policies learned in simulation to the real world is a promising strategy for acquiring robot skills at scale.
In this paper, we investigate using Large Language Models (LLMs) to automate and accelerate sim-to-real design.
Our approach is capable of solving novel robot tasks, such as quadruped balancing and walking atop a yoga ball.
arXiv Detail & Related papers (2024-06-04T04:53:05Z) - Robust Visual Sim-to-Real Transfer for Robotic Manipulation [79.66851068682779]
Learning visuomotor policies in simulation is much safer and cheaper than in the real world.
However, due to discrepancies between the simulated and real data, simulator-trained policies often fail when transferred to real robots.
One common approach to bridge the visual sim-to-real domain gap is domain randomization (DR)
arXiv Detail & Related papers (2023-07-28T05:47:24Z) - Provable Sim-to-real Transfer in Continuous Domain with Partial
Observations [39.18274543757048]
Sim-to-real transfer trains RL agents in the simulated environments and then deploys them in the real world.
We show that a popular robust adversarial training algorithm is capable of learning a policy from the simulated environment that is competitive to the optimal policy in the real-world environment.
arXiv Detail & Related papers (2022-10-27T16:37:52Z) - Auto-Tuned Sim-to-Real Transfer [143.44593793640814]
Policies trained in simulation often fail when transferred to the real world.
Current approaches to tackle this problem, such as domain randomization, require prior knowledge and engineering.
We propose a method for automatically tuning simulator system parameters to match the real world.
arXiv Detail & Related papers (2021-04-15T17:59:55Z) - TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors [74.67698916175614]
We propose TrafficSim, a multi-agent behavior model for realistic traffic simulation.
In particular, we leverage an implicit latent variable model to parameterize a joint actor policy.
We show TrafficSim generates significantly more realistic and diverse traffic scenarios as compared to a diverse set of baselines.
arXiv Detail & Related papers (2021-01-17T00:29:30Z) - Point Cloud Based Reinforcement Learning for Sim-to-Real and Partial
Observability in Visual Navigation [62.22058066456076]
Reinforcement Learning (RL) represents powerful tools to solve complex robotic tasks.
RL does not work directly in the real-world, which is known as the sim-to-real transfer problem.
We propose a method that learns on an observation space constructed by point clouds and environment randomization.
arXiv Detail & Related papers (2020-07-27T17:46:59Z) - Data-efficient Domain Randomization with Bayesian Optimization [34.854609756970305]
When learning policies for robot control, the required real-world data is typically prohibitively expensive to acquire.
BayRn is a black-box sim-to-real algorithm that solves tasks efficiently by adapting the domain parameter distribution.
Our results show that BayRn is able to perform sim-to-real transfer, while significantly reducing the required prior knowledge.
arXiv Detail & Related papers (2020-03-05T07:48:31Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.