BayRnTune: Adaptive Bayesian Domain Randomization via Strategic Fine-tuning

• URL: http://arxiv.org/abs/2310.10606v1
• Date: Mon, 16 Oct 2023 17:32:23 GMT
• Title: BayRnTune: Adaptive Bayesian Domain Randomization via Strategic Fine-tuning
• Authors: Tianle Huang, Nitish Sontakke, K. Niranjan Kumar, Irfan Essa, Stefanos Nikolaidis, Dennis W. Hong, Sehoon Ha
• Abstract summary: Domain randomization (DR) entails training a policy with randomized dynamics. BayRnTune aims to significantly accelerate the learning processes by fine-tuning from previously learned policy.
• Score: 30.753772054098526
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Domain randomization (DR), which entails training a policy with randomized dynamics, has proven to be a simple yet effective algorithm for reducing the gap between simulation and the real world. However, DR often requires careful tuning of randomization parameters. Methods like Bayesian Domain Randomization (Bayesian DR) and Active Domain Randomization (Adaptive DR) address this issue by automating parameter range selection using real-world experience. While effective, these algorithms often require long computation time, as a new policy is trained from scratch every iteration. In this work, we propose Adaptive Bayesian Domain Randomization via Strategic Fine-tuning (BayRnTune), which inherits the spirit of BayRn but aims to significantly accelerate the learning processes by fine-tuning from previously learned policy. This idea leads to a critical question: which previous policy should we use as a prior during fine-tuning? We investigated four different fine-tuning strategies and compared them against baseline algorithms in five simulated environments, ranging from simple benchmark tasks to more complex legged robot environments. Our analysis demonstrates that our method yields better rewards in the same amount of timesteps compared to vanilla domain randomization or Bayesian DR.

Related papers

• Thompson sampling for improved exploration in GFlowNets [75.89693358516944]
  Generative flow networks (GFlowNets) are amortized variational inference algorithms that treat sampling from a distribution over compositional objects as a sequential decision-making problem with a learnable action policy. We show in two domains that TS-GFN yields improved exploration and thus faster convergence to the target distribution than the off-policy exploration strategies used in past work.
  arXiv  Detail & Related papers  (2023-06-30T14:19:44Z)
• Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guarantees [56.848265937921354]
  Inverse reinforcement learning (IRL) aims to recover the reward function and the associated optimal policy. Many algorithms for IRL have an inherently nested structure. We develop a novel single-loop algorithm for IRL that does not compromise reward estimation accuracy.
  arXiv  Detail & Related papers  (2022-10-04T17:13:45Z)
• Cyclic Policy Distillation: Sample-Efficient Sim-to-Real Reinforcement Learning with Domain Randomization [10.789649934346004]
  We propose a sample-efficient method named cyclic policy distillation (CPD) CPD divides the range of randomized parameters into several small sub-domains and assigns a local policy to each one. All of the learned local policies are distilled into a global policy for sim-to-real transfers.
  arXiv  Detail & Related papers  (2022-07-29T09:22:53Z)
• Dimensionality Reduction and Prioritized Exploration for Policy Search [29.310742141970394]
  Black-box policy optimization is a class of reinforcement learning algorithms that explores and updates the policies at the parameter level. We present a novel method to prioritize the exploration of effective parameters and cope with full covariance matrix updates. Our algorithm learns faster than recent approaches and requires fewer samples to achieve state-of-the-art results.
  arXiv  Detail & Related papers  (2022-03-09T15:17:09Z)
• Bregman Gradient Policy Optimization [97.73041344738117]
  We design a Bregman gradient policy optimization for reinforcement learning based on Bregman divergences and momentum techniques. VR-BGPO reaches the best complexity $tilde(epsilon-3)$ for finding an $epsilon$stationary point only requiring one trajectory at each iteration.
  arXiv  Detail & Related papers  (2021-06-23T01:08:54Z)
• Local policy search with Bayesian optimization [73.0364959221845]
  Reinforcement learning aims to find an optimal policy by interaction with an environment. Policy gradients for local search are often obtained from random perturbations. We develop an algorithm utilizing a probabilistic model of the objective function and its gradient.
  arXiv  Detail & Related papers  (2021-06-22T16:07:02Z)
• Learning Sampling Policy for Faster Derivative Free Optimization [100.27518340593284]
  We propose a new reinforcement learning based ZO algorithm (ZO-RL) with learning the sampling policy for generating the perturbations in ZO optimization instead of using random sampling. Our results show that our ZO-RL algorithm can effectively reduce the variances of ZO gradient by learning a sampling policy, and converge faster than existing ZO algorithms in different scenarios.
  arXiv  Detail & Related papers  (2021-04-09T14:50:59Z)
• Variance Penalized On-Policy and Off-Policy Actor-Critic [60.06593931848165]
  We propose on-policy and off-policy actor-critic algorithms that optimize a performance criterion involving both mean and variance in the return. Our approach not only performs on par with actor-critic and prior variance-penalization baselines in terms of expected return, but also generates trajectories which have lower variance in the return.
  arXiv  Detail & Related papers  (2021-02-03T10:06:16Z)
• Policy Transfer via Kinematic Domain Randomization and Adaptation [22.038635244802798]
  We investigate the impact of randomized parameter selection on policy transferability across different types of domain discrepancies. We introduce a new domain adaptation algorithm that utilizes simulated kinematic parameters variation. We showcase our findings on a simulated quadruped robot in five different target environments.
  arXiv  Detail & Related papers  (2020-11-03T18:09:35Z)
• Data-efficient Domain Randomization with Bayesian Optimization [34.854609756970305]
  When learning policies for robot control, the required real-world data is typically prohibitively expensive to acquire. BayRn is a black-box sim-to-real algorithm that solves tasks efficiently by adapting the domain parameter distribution. Our results show that BayRn is able to perform sim-to-real transfer, while significantly reducing the required prior knowledge.
  arXiv  Detail & Related papers  (2020-03-05T07:48:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.