Robust Reinforcement Learning via Adversarial training with Langevin Dynamics

• URL: http://arxiv.org/abs/2002.06063v2
• Date: Thu, 5 Nov 2020 19:09:36 GMT
• Title: Robust Reinforcement Learning via Adversarial training with Langevin Dynamics
• Authors: Parameswaran Kamalaruban, Yu-Ting Huang, Ya-Ping Hsieh, Paul Rolland, Cheng Shi, Volkan Cevher
• Abstract summary: We introduce a sampling perspective to tackle the challenging task of training robust Reinforcement Learning (RL) agents. We present a novel, scalable two-player RL algorithm, which is a sampling variant of the two-player policy method.
• Score: 51.234482917047835
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce a sampling perspective to tackle the challenging task of training robust Reinforcement Learning (RL) agents. Leveraging the powerful Stochastic Gradient Langevin Dynamics, we present a novel, scalable two-player RL algorithm, which is a sampling variant of the two-player policy gradient method. Our algorithm consistently outperforms existing baselines, in terms of generalization across different training and testing conditions, on several MuJoCo environments. Our experiments also show that, even for objective functions that entirely ignore potential environmental shifts, our sampling approach remains highly robust in comparison to standard RL algorithms.

Related papers

• ODRL: A Benchmark for Off-Dynamics Reinforcement Learning [59.72217833812439]
  We introduce ODRL, the first benchmark tailored for evaluating off-dynamics RL methods. ODRL contains four experimental settings where the source and target domains can be either online or offline. We conduct extensive benchmarking experiments, which show that no method has universal advantages across varied dynamics shifts.
  arXiv  Detail & Related papers  (2024-10-28T05:29:38Z)
• Multi-turn Reinforcement Learning from Preference Human Feedback [41.327438095745315]
  Reinforcement Learning from Human Feedback (RLHF) has become the standard approach for aligning Large Language Models with human preferences. Existing methods work by emulating the preferences at the single decision (turn) level. We develop novel methods for Reinforcement Learning from preference feedback between two full multi-turn conversations.
  arXiv  Detail & Related papers  (2024-05-23T14:53:54Z)
• Natural Actor-Critic for Robust Reinforcement Learning with Function Approximation [20.43657369407846]
  We study robust reinforcement learning (RL) with the goal of determining a well-performing policy that is robust against model mismatch between the training simulator and the testing environment. We propose two novel uncertainty set formulations, one based on double sampling and the other on an integral probability metric. We demonstrate the robust performance of the policy learned by our proposed RNAC approach in multiple MuJoCo environments and a real-world TurtleBot navigation task.
  arXiv  Detail & Related papers  (2023-07-17T22:10:20Z)
• Proximal Policy Gradient Arborescence for Quality Diversity Reinforcement Learning [14.16864939687988]
  Training generally capable agents that thoroughly explore their environment and learn new and diverse skills is a long-term goal of robot learning. Quality Diversity Reinforcement Learning (QD-RL) is an emerging research area that blends the best aspects of both fields.
  arXiv  Detail & Related papers  (2023-05-23T08:05:59Z)
• Predictive Experience Replay for Continual Visual Control and Forecasting [62.06183102362871]
  We present a new continual learning approach for visual dynamics modeling and explore its efficacy in visual control and forecasting. We first propose the mixture world model that learns task-specific dynamics priors with a mixture of Gaussians, and then introduce a new training strategy to overcome catastrophic forgetting. Our model remarkably outperforms the naive combinations of existing continual learning and visual RL algorithms on DeepMind Control and Meta-World benchmarks with continual visual control tasks.
  arXiv  Detail & Related papers  (2023-03-12T05:08:03Z)
• Learning Dynamics and Generalization in Reinforcement Learning [59.530058000689884]
  We show theoretically that temporal difference learning encourages agents to fit non-smooth components of the value function early in training. We show that neural networks trained using temporal difference algorithms on dense reward tasks exhibit weaker generalization between states than randomly networks and gradient networks trained with policy methods.
  arXiv  Detail & Related papers  (2022-06-05T08:49:16Z)
• Learning to Continuously Optimize Wireless Resource in a Dynamic Environment: A Bilevel Optimization Perspective [52.497514255040514]
  This work develops a new approach that enables data-driven methods to continuously learn and optimize resource allocation strategies in a dynamic environment. We propose to build the notion of continual learning into wireless system design, so that the learning model can incrementally adapt to the new episodes. Our design is based on a novel bilevel optimization formulation which ensures certain fairness" across different data samples.
  arXiv  Detail & Related papers  (2021-05-03T07:23:39Z)
• SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning [102.78958681141577]
  We present SUNRISE, a simple unified ensemble method, which is compatible with various off-policy deep reinforcement learning algorithms. SUNRISE integrates two key ingredients: (a) ensemble-based weighted Bellman backups, which re-weight target Q-values based on uncertainty estimates from a Q-ensemble, and (b) an inference method that selects actions using the highest upper-confidence bounds for efficient exploration.
  arXiv  Detail & Related papers  (2020-07-09T17:08:44Z)
• Accelerating Reinforcement Learning with a Directional-Gaussian-Smoothing Evolution Strategy [3.404507240556492]
  Evolution strategy (ES) has been shown great promise in many challenging reinforcement learning (RL) tasks. There are two limitations in the current ES practice that may hinder its otherwise further capabilities. In this work, we employ a Directional Gaussian Smoothing Evolutionary Strategy (DGS-ES) to accelerate RL training. We show that DGS-ES is highly scalable, possesses superior wall-clock time, and achieves competitive reward scores to other popular policy gradient and ES approaches.
  arXiv  Detail & Related papers  (2020-02-21T01:05:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.