Competitiveness of MAP-Elites against Proximal Policy Optimization on locomotion tasks in deterministic simulations

• URL: http://arxiv.org/abs/2009.08438v2
• Date: Sat, 19 Sep 2020 08:33:45 GMT
• Title: Competitiveness of MAP-Elites against Proximal Policy Optimization on locomotion tasks in deterministic simulations
• Authors: Szymon Brych and Antoine Cully
• Abstract summary: We show that Multidimensional Archive of Phenotypic Elites (MAP-Elites) can deliver better-performing solutions than one of the state-of-the-art RL methods. This paper demonstrates that EAs combined with modern computational resources display promising characteristics.
• Score: 1.827510863075184
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The increasing importance of robots and automation creates a demand for learnable controllers which can be obtained through various approaches such as Evolutionary Algorithms (EAs) or Reinforcement Learning (RL). Unfortunately, these two families of algorithms have mainly developed independently and there are only a few works comparing modern EAs with deep RL algorithms. We show that Multidimensional Archive of Phenotypic Elites (MAP-Elites), which is a modern EA, can deliver better-performing solutions than one of the state-of-the-art RL methods, Proximal Policy Optimization (PPO) in the generation of locomotion controllers for a simulated hexapod robot. Additionally, extensive hyper-parameter tuning shows that MAP-Elites displays greater robustness across seeds and hyper-parameter sets. Generally, this paper demonstrates that EAs combined with modern computational resources display promising characteristics and have the potential to contribute to the state-of-the-art in controller learning.

Related papers

• SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learning [85.21378553454672]
  We develop a library containing a sample efficient off-policy deep RL method, together with methods for computing rewards and resetting the environment. We find that our implementation can achieve very efficient learning, acquiring policies for PCB board assembly, cable routing, and object relocation. These policies achieve perfect or near-perfect success rates, extreme robustness even under perturbations, and exhibit emergent robustness recovery and correction behaviors.
  arXiv  Detail & Related papers  (2024-01-29T10:01:10Z)
• Machine Learning Insides OptVerse AI Solver: Design Principles and Applications [74.67495900436728]
  We present a comprehensive study on the integration of machine learning (ML) techniques into Huawei Cloud's OptVerse AI solver. We showcase our methods for generating complex SAT and MILP instances utilizing generative models that mirror multifaceted structures of real-world problem. We detail the incorporation of state-of-the-art parameter tuning algorithms which markedly elevate solver performance.
  arXiv  Detail & Related papers  (2024-01-11T15:02:15Z)
• Waymax: An Accelerated, Data-Driven Simulator for Large-Scale Autonomous Driving Research [76.93956925360638]
  Waymax is a new data-driven simulator for autonomous driving in multi-agent scenes. It runs entirely on hardware accelerators such as TPUs/GPUs and supports in-graph simulation for training. We benchmark a suite of popular imitation and reinforcement learning algorithms with ablation studies on different design decisions.
  arXiv  Detail & Related papers  (2023-10-12T20:49:15Z)
• Reinforcement Learning-assisted Evolutionary Algorithm: A Survey and Research Opportunities [63.258517066104446]
  Reinforcement learning integrated as a component in the evolutionary algorithm has demonstrated superior performance in recent years. We discuss the RL-EA integration method, the RL-assisted strategy adopted by RL-EA, and its applications according to the existing literature. In the applications of RL-EA section, we also demonstrate the excellent performance of RL-EA on several benchmarks and a range of public datasets.
  arXiv  Detail & Related papers  (2023-08-25T15:06:05Z)
• Karolos: An Open-Source Reinforcement Learning Framework for Robot-Task Environments [0.3867363075280544]
  In reinforcement learning (RL) research, simulations enable benchmarks between algorithms. In this paper, we introduce Karolos, a framework developed for robotic applications. The code is open source and published on GitHub with the aim of promoting research of RL applications in robotics.
  arXiv  Detail & Related papers  (2022-12-01T23:14:02Z)
• Training Diverse High-Dimensional Controllers by Scaling Covariance Matrix Adaptation MAP-Annealing [12.90845054806193]
  Pre-training a diverse set of neural network controllers in simulation has enabled robots to adapt online to damage in robot locomotion tasks. CMA-MAE, an evolution strategies (ES)-based quality diversity algorithm, does not have these limitations and has achieved state-of-the-art performance on standard QD benchmarks. We propose three new CMA-MAE variants that scale to high dimensions.
  arXiv  Detail & Related papers  (2022-10-06T01:03:01Z)
• Bayesian Generational Population-Based Training [35.70338636901159]
  Population-Based Training (PBT) has led to impressive performance in several large scale settings. We introduce two new innovations in PBT-style methods. We show that these innovations lead to large performance gains.
  arXiv  Detail & Related papers  (2022-07-19T16:57:38Z)
• MAMBPO: Sample-efficient multi-robot reinforcement learning using learned world models [4.84279798426797]
  Multi-robot systems can benefit from reinforcement learning (RL) algorithms that learn behaviours in a small number of trials. We present a novel multi-agent model-based RL algorithm: Multi-Agent Model-Based Policy Optimization (MAMBPO)
  arXiv  Detail & Related papers  (2021-03-05T13:37:23Z)
• Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planning [93.1435980666675]
  We show how optimistic exploration can be easily combined with state-of-the-art reinforcement learning algorithms. Our experiments demonstrate that optimistic exploration significantly speeds-up learning when there are penalties on actions.
  arXiv  Detail & Related papers  (2020-06-15T18:37:38Z)
• The Adversarial Resilience Learning Architecture for AI-based Modelling, Exploration, and Operation of Complex Cyber-Physical Systems [0.0]
  We describe the concept of Adversarial Learning (ARL) that formulates a new approach to complex environment checking and resilient operation. The quintessence of ARL lies in both agents exploring the system and training each other without any domain knowledge. Here, we introduce the ARL software architecture that allows to use a wide range of model-free as well as model-based DRL-based algorithms.
  arXiv  Detail & Related papers  (2020-05-27T19:19:57Z)
• Information Theoretic Model Predictive Q-Learning [64.74041985237105]
  We present a novel theoretical connection between information theoretic MPC and entropy regularized RL. We develop a Q-learning algorithm that can leverage biased models.
  arXiv  Detail & Related papers  (2019-12-31T00:29:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.