Distributionally Robust Policy Learning via Adversarial Environment Generation

• URL: http://arxiv.org/abs/2107.06353v1
• Date: Tue, 13 Jul 2021 19:26:34 GMT
• Title: Distributionally Robust Policy Learning via Adversarial Environment Generation
• Authors: Allen Z. Ren, Anirudha Majumdar
• Abstract summary: We propose DRAGEN - Distributionally Robust policy learning via Adversarial Generation of ENvironments. We learn a generative model for environments whose latent variables capture cost-predictive and realistic variations in environments. We demonstrate strong Out-of-Distribution (OoD) generalization in simulation for grasping realistic 2D/3D objects.
• Score: 3.42658286826597
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Our goal is to train control policies that generalize well to unseen environments. Inspired by the Distributionally Robust Optimization (DRO) framework, we propose DRAGEN - Distributionally Robust policy learning via Adversarial Generation of ENvironments - for iteratively improving robustness of policies to realistic distribution shifts by generating adversarial environments. The key idea is to learn a generative model for environments whose latent variables capture cost-predictive and realistic variations in environments. We perform DRO with respect to a Wasserstein ball around the empirical distribution of environments by generating realistic adversarial environments via gradient ascent on the latent space. We demonstrate strong Out-of-Distribution (OoD) generalization in simulation for (i) swinging up a pendulum with onboard vision and (ii) grasping realistic 2D/3D objects. Grasping experiments on hardware demonstrate better sim2real performance compared to domain randomization.

Related papers

• LoopSR: Looping Sim-and-Real for Lifelong Policy Adaptation of Legged Robots [20.715834172041763]
  We propose a lifelong policy adaptation framework named LoopSR. It reconstructs the real-world environments back in simulation for further improvement. By leveraging the continual training, LoopSR achieves superior data efficiency compared with strong baselines.
  arXiv  Detail & Related papers  (2024-09-26T16:02:25Z)
• Evaluating Real-World Robot Manipulation Policies in Simulation [91.55267186958892]
  Control and visual disparities between real and simulated environments are key challenges for reliable simulated evaluation. We propose approaches for mitigating these gaps without needing to craft full-fidelity digital twins of real-world environments. We create SIMPLER, a collection of simulated environments for manipulation policy evaluation on common real robot setups.
  arXiv  Detail & Related papers  (2024-05-09T17:30:16Z)
• A Conservative Approach for Few-Shot Transfer in Off-Dynamics Reinforcement Learning [3.1515473193934778]
  Off-dynamics Reinforcement Learning seeks to transfer a policy from a source environment to a target environment characterized by distinct yet similar dynamics. We propose an innovative approach inspired by recent advancements in Imitation Learning and conservative RL algorithms.
  arXiv  Detail & Related papers  (2023-12-24T13:09:08Z)
• Robust Visual Sim-to-Real Transfer for Robotic Manipulation [79.66851068682779]
  Learning visuomotor policies in simulation is much safer and cheaper than in the real world. However, due to discrepancies between the simulated and real data, simulator-trained policies often fail when transferred to real robots. One common approach to bridge the visual sim-to-real domain gap is domain randomization (DR)
  arXiv  Detail & Related papers  (2023-07-28T05:47:24Z)
• Reparameterized Policy Learning for Multimodal Trajectory Optimization [61.13228961771765]
  We investigate the challenge of parametrizing policies for reinforcement learning in high-dimensional continuous action spaces. We propose a principled framework that models the continuous RL policy as a generative model of optimal trajectories. We present a practical model-based RL method, which leverages the multimodal policy parameterization and learned world model.
  arXiv  Detail & Related papers  (2023-07-20T09:05:46Z)
• Grounding Aleatoric Uncertainty in Unsupervised Environment Design [32.00797965770773]
  In partially-observable settings, optimal policies may depend on the ground-truth distribution over aleatoric parameters of the environment. We propose a minimax regret UED method that optimize the ground-truth utility function, even when the underlying training data is biased due to CICS.
  arXiv  Detail & Related papers  (2022-07-11T22:45:29Z)
• Uncertainty Aware System Identification with Universal Policies [45.44896435487879]
  Sim2real transfer is concerned with transferring policies trained in simulation to potentially noisy real world environments. We propose Uncertainty-aware policy search (UncAPS), where we use Universal Policy Network (UPN) to store simulation-trained task-specific policies. We then employ robust Bayesian optimisation to craft robust policies for the given environment by combining relevant UPN policies in a DR like fashion.
  arXiv  Detail & Related papers  (2022-02-11T18:27:23Z)
• Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design [121.73425076217471]
  We propose Unsupervised Environment Design (UED), where developers provide environments with unknown parameters, and these parameters are used to automatically produce a distribution over valid, solvable environments. We call our technique Protagonist Antagonist Induced Regret Environment Design (PAIRED) Our experiments demonstrate that PAIRED produces a natural curriculum of increasingly complex environments, and PAIRED agents achieve higher zero-shot transfer performance when tested in highly novel environments.
  arXiv  Detail & Related papers  (2020-12-03T17:37:01Z)
• Implicit Distributional Reinforcement Learning [61.166030238490634]
  implicit distributional actor-critic (IDAC) built on two deep generator networks (DGNs) Semi-implicit actor (SIA) powered by a flexible policy distribution. We observe IDAC outperforms state-of-the-art algorithms on representative OpenAI Gym environments.
  arXiv  Detail & Related papers  (2020-07-13T02:52:18Z)
• Environment Shaping in Reinforcement Learning using State Abstraction [63.444831173608605]
  We propose a novel framework of emphenvironment shaping using state abstraction. Our key idea is to compress the environment's large state space with noisy signals to an abstracted space. We show that the agent's policy learnt in the shaped environment preserves near-optimal behavior in the original environment.
  arXiv  Detail & Related papers  (2020-06-23T17:00:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.