Training Diverse High-Dimensional Controllers by Scaling Covariance
Matrix Adaptation MAP-Annealing
- URL: http://arxiv.org/abs/2210.02622v3
- Date: Sat, 16 Sep 2023 02:17:57 GMT
- Title: Training Diverse High-Dimensional Controllers by Scaling Covariance
Matrix Adaptation MAP-Annealing
- Authors: Bryon Tjanaka, Matthew C. Fontaine, David H. Lee, Aniruddha Kalkar,
Stefanos Nikolaidis
- Abstract summary: Pre-training a diverse set of neural network controllers in simulation has enabled robots to adapt online to damage in robot locomotion tasks.
CMA-MAE, an evolution strategies (ES)-based quality diversity algorithm, does not have these limitations and has achieved state-of-the-art performance on standard QD benchmarks.
We propose three new CMA-MAE variants that scale to high dimensions.
- Score: 12.90845054806193
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Pre-training a diverse set of neural network controllers in simulation has
enabled robots to adapt online to damage in robot locomotion tasks. However,
finding diverse, high-performing controllers requires expensive network
training and extensive tuning of a large number of hyperparameters. On the
other hand, Covariance Matrix Adaptation MAP-Annealing (CMA-MAE), an evolution
strategies (ES)-based quality diversity algorithm, does not have these
limitations and has achieved state-of-the-art performance on standard QD
benchmarks. However, CMA-MAE cannot scale to modern neural network controllers
due to its quadratic complexity. We leverage efficient approximation methods in
ES to propose three new CMA-MAE variants that scale to high dimensions. Our
experiments show that the variants outperform ES-based baselines in benchmark
robotic locomotion tasks, while being comparable with or exceeding
state-of-the-art deep reinforcement learning-based quality diversity
algorithms.
Related papers
- RLEEGNet: Integrating Brain-Computer Interfaces with Adaptive AI for
Intuitive Responsiveness and High-Accuracy Motor Imagery Classification [0.0]
We introduce a framework that leverages Reinforcement Learning with Deep Q-Networks (DQN) for classification tasks.
We present a preprocessing technique for multiclass motor imagery (MI) classification in a One-Versus-The-Rest (OVR) manner.
The integration of DQN with a 1D-CNN-LSTM architecture optimize the decision-making process in real-time.
arXiv Detail & Related papers (2024-02-09T02:03:13Z) - A Multi-Head Ensemble Multi-Task Learning Approach for Dynamical
Computation Offloading [62.34538208323411]
We propose a multi-head ensemble multi-task learning (MEMTL) approach with a shared backbone and multiple prediction heads (PHs)
MEMTL outperforms benchmark methods in both the inference accuracy and mean square error without requiring additional training data.
arXiv Detail & Related papers (2023-09-02T11:01:16Z) - Complex Locomotion Skill Learning via Differentiable Physics [30.868690308658174]
Differentiable physics enables efficient-based optimizations of neural network (NN) controllers.
We present a practical learning framework that outputs unified NN controllers capable of tasks with significantly improved complexity and diversity.
arXiv Detail & Related papers (2022-06-06T04:01:12Z) - Intelligent Trajectory Design for RIS-NOMA aided Multi-robot
Communications [59.34642007625687]
The goal is to maximize the sum-rate of whole trajectories for multi-robot system by jointly optimizing trajectories and NOMA decoding orders of robots.
An integrated machine learning (ML) scheme is proposed, which combines long short-term memory (LSTM)-autoregressive integrated moving average (ARIMA) model and dueling double deep Q-network (D$3$QN) algorithm.
arXiv Detail & Related papers (2022-05-03T17:14:47Z) - Collaborative Intelligent Reflecting Surface Networks with Multi-Agent
Reinforcement Learning [63.83425382922157]
Intelligent reflecting surface (IRS) is envisioned to be widely applied in future wireless networks.
In this paper, we investigate a multi-user communication system assisted by cooperative IRS devices with the capability of energy harvesting.
arXiv Detail & Related papers (2022-03-26T20:37:14Z) - Hierarchical Multi-Agent DRL-Based Framework for Joint Multi-RAT
Assignment and Dynamic Resource Allocation in Next-Generation HetNets [21.637440368520487]
This paper considers the problem of cost-aware downlink sum-rate via joint optimal radio access technologies (RATs) assignment and power allocation in next-generation wireless networks (HetNets)
We propose a hierarchical multi-agent deep reinforcement learning (DRL) framework, called DeepRAT, to solve it efficiently and learn system dynamics.
In particular, the DeepRAT framework decomposes the problem into two main stages; the RATs-EDs assignment stage, which implements a single-agent Deep Q Network algorithm, and the power allocation stage, which utilizes a multi-agent Deep Deterministic Policy Gradient
arXiv Detail & Related papers (2022-02-28T09:49:44Z) - Adaptive Anomaly Detection for Internet of Things in Hierarchical Edge
Computing: A Contextual-Bandit Approach [81.5261621619557]
We propose an adaptive anomaly detection scheme with hierarchical edge computing (HEC)
We first construct multiple anomaly detection DNN models with increasing complexity, and associate each of them to a corresponding HEC layer.
Then, we design an adaptive model selection scheme that is formulated as a contextual-bandit problem and solved by using a reinforcement learning policy network.
arXiv Detail & Related papers (2021-08-09T08:45:47Z) - Competitiveness of MAP-Elites against Proximal Policy Optimization on
locomotion tasks in deterministic simulations [1.827510863075184]
We show that Multidimensional Archive of Phenotypic Elites (MAP-Elites) can deliver better-performing solutions than one of the state-of-the-art RL methods.
This paper demonstrates that EAs combined with modern computational resources display promising characteristics.
arXiv Detail & Related papers (2020-09-17T17:41:46Z) - Deep Multi-Task Learning for Cooperative NOMA: System Design and
Principles [52.79089414630366]
We develop a novel deep cooperative NOMA scheme, drawing upon the recent advances in deep learning (DL)
We develop a novel hybrid-cascaded deep neural network (DNN) architecture such that the entire system can be optimized in a holistic manner.
arXiv Detail & Related papers (2020-07-27T12:38:37Z) - Communication-Efficient Distributed Stochastic AUC Maximization with
Deep Neural Networks [50.42141893913188]
We study a distributed variable for large-scale AUC for a neural network as with a deep neural network.
Our model requires a much less number of communication rounds and still a number of communication rounds in theory.
Our experiments on several datasets show the effectiveness of our theory and also confirm our theory.
arXiv Detail & Related papers (2020-05-05T18:08:23Z) - Scaling MAP-Elites to Deep Neuroevolution [5.332714036560255]
We propose to leverage the efficiency of Evolution Strategies (ES) to scale MAP-Elites to high-dimensional controllers parameterized by large neural networks.
We show that ME-ES performs efficient exploration, on par with state-of-the-art exploration algorithms in high-dimensional control tasks with strongly deceptive rewards.
arXiv Detail & Related papers (2020-03-03T23:02:37Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.