Continuous Control With Ensemble Deep Deterministic Policy Gradients

• URL: http://arxiv.org/abs/2111.15382v1
• Date: Tue, 30 Nov 2021 13:28:13 GMT
• Title: Continuous Control With Ensemble Deep Deterministic Policy Gradients
• Abstract summary: We show how Ensemble Deep Deterministic Policy Gradients (ED2) can yield state-of-the-art results on continuous control tasks from OpenAI Gym MuJoCo. From the practical side, ED2 is conceptually straightforward, easy to code, and does not require knowledge outside of the existing deep reinforcement learning toolbox.
• Score: 3.3465375629622502
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The growth of deep reinforcement learning (RL) has brought multiple exciting tools and methods to the field. This rapid expansion makes it important to understand the interplay between individual elements of the RL toolbox. We approach this task from an empirical perspective by conducting a study in the continuous control setting. We present multiple insights of fundamental nature, including: an average of multiple actors trained from the same data boosts performance; the existing methods are unstable across training runs, epochs of training, and evaluation runs; a commonly used additive action noise is not required for effective training; a strategy based on posterior sampling explores better than the approximated UCB combined with the weighted Bellman backup; the weighted Bellman backup alone cannot replace the clipped double Q-Learning; the critics' initialization plays the major role in ensemble-based actor-critic exploration. As a conclusion, we show how existing tools can be brought together in a novel way, giving rise to the Ensemble Deep Deterministic Policy Gradients (ED2) method, to yield state-of-the-art results on continuous control tasks from OpenAI Gym MuJoCo. From the practical side, ED2 is conceptually straightforward, easy to code, and does not require knowledge outside of the existing RL toolbox.

Related papers

• ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning [75.73135757250806]
  Agentic reinforcement learning (ARL) has rapidly gained attention as a promising paradigm for training agents to solve complex, multi-step interactive tasks.<n>Despite encouraging early results, ARL remains highly unstable, often leading to training collapse.<n>In this paper, we first propose ARLArena, a stable training recipe and systematic analysis framework that examines training stability in a controlled and reproducible setting.
  arXiv  Detail & Related papers  (2026-02-25T03:43:34Z)
• On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models [73.10315509190623]
  Recent reinforcement learning techniques have yielded impressive reasoning improvements in language models.<n>It remains unclear whether post-training truly extends a model's reasoning ability beyond what it acquires during pre-training.<n>We develop a fully controlled experimental framework that isolates the causal contributions of pre-training, mid-training, and RL-based post-training.
  arXiv  Detail & Related papers  (2025-12-08T18:12:10Z)
• RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning [125.65034908728828]
  Training large language models (LLMs) as interactive agents presents unique challenges. While reinforcement learning has enabled progress in static tasks, multi-turn agent RL training remains underexplored. We propose StarPO, a general framework for trajectory-level agent RL, and introduce RAGEN, a modular system for training and evaluating LLM agents.
  arXiv  Detail & Related papers  (2025-04-24T17:57:08Z)
• Reinforcement Learning with Action Sequence for Data-Efficient Robot Learning [62.3886343725955]
  We introduce a novel RL algorithm that learns a critic network that outputs Q-values over a sequence of actions. By explicitly training the value functions to learn the consequence of executing a series of current and future actions, our algorithm allows for learning useful value functions from noisy trajectories.
  arXiv  Detail & Related papers  (2024-11-19T01:23:52Z)
• Two-stage Learning-to-Defer for Multi-Task Learning [3.4289478404209826]
  We introduce a Learning-to-Defer approach for multi-task learning that encompasses both classification and regression tasks. Our two-stage approach utilizes a rejector that defers decisions to the most accurate agent among a pre-trained joint-regressor models and one or more external experts.
  arXiv  Detail & Related papers  (2024-10-21T07:44:57Z)
• Exploiting Estimation Bias in Clipped Double Q-Learning for Continous Control Reinforcement Learning Tasks [5.968716050740402]
  This paper focuses on addressing and exploiting estimation biases in Actor-Critic methods for continuous control tasks. We design a Bias Exploiting (BE) mechanism to dynamically select the most advantageous estimation bias during training of the RL agent. Most State-of-the-art Deep RL algorithms can be equipped with the BE mechanism, without hindering performance or computational complexity.
  arXiv  Detail & Related papers  (2024-02-14T10:44:03Z)
• Demonstration-free Autonomous Reinforcement Learning via Implicit and Bidirectional Curriculum [22.32327908453603]
  We propose a demonstration-free reinforcement learning algorithm via Implicit and Bi-directional Curriculum (IBC) With an auxiliary agent that is conditionally activated upon learning progress and a bidirectional goal curriculum based on optimal transport, our method outperforms previous methods.
  arXiv  Detail & Related papers  (2023-05-17T04:31:36Z)
• Jump-Start Reinforcement Learning [68.82380421479675]
  We present a meta algorithm that can use offline data, demonstrations, or a pre-existing policy to initialize an RL policy. In particular, we propose Jump-Start Reinforcement Learning (JSRL), an algorithm that employs two policies to solve tasks. We show via experiments that JSRL is able to significantly outperform existing imitation and reinforcement learning algorithms.
  arXiv  Detail & Related papers  (2022-04-05T17:25:22Z)
• URLB: Unsupervised Reinforcement Learning Benchmark [82.36060735454647]
  We introduce the Unsupervised Reinforcement Learning Benchmark (URLB) URLB consists of two phases: reward-free pre-training and downstream task adaptation with extrinsic rewards. We provide twelve continuous control tasks from three domains for evaluation and open-source code for eight leading unsupervised RL methods.
  arXiv  Detail & Related papers  (2021-10-28T15:07:01Z)
• PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training [94.87393610927812]
  We present an off-policy, interactive reinforcement learning algorithm that capitalizes on the strengths of both feedback and off-policy learning. We demonstrate that our approach is capable of learning tasks of higher complexity than previously considered by human-in-the-loop methods.
  arXiv  Detail & Related papers  (2021-06-09T14:10:50Z)
• SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning [102.78958681141577]
  We present SUNRISE, a simple unified ensemble method, which is compatible with various off-policy deep reinforcement learning algorithms. SUNRISE integrates two key ingredients: (a) ensemble-based weighted Bellman backups, which re-weight target Q-values based on uncertainty estimates from a Q-ensemble, and (b) an inference method that selects actions using the highest upper-confidence bounds for efficient exploration.
  arXiv  Detail & Related papers  (2020-07-09T17:08:44Z)
• Robust Reinforcement Learning via Adversarial training with Langevin Dynamics [51.234482917047835]
  We introduce a sampling perspective to tackle the challenging task of training robust Reinforcement Learning (RL) agents. We present a novel, scalable two-player RL algorithm, which is a sampling variant of the two-player policy method.
  arXiv  Detail & Related papers  (2020-02-14T14:59:14Z)
• Ready Policy One: World Building Through Active Learning [35.358315617358976]
  We introduce Ready Policy One (RP1), a framework that views Model-Based Reinforcement Learning as an active learning problem. RP1 achieves this by utilizing a hybrid objective function, which crucially adapts during optimization. We rigorously evaluate our method on a variety of continuous control tasks, and demonstrate statistically significant gains over existing approaches.
  arXiv  Detail & Related papers  (2020-02-07T09:57:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.