Behavior-based Neuroevolutionary Training in Reinforcement Learning

• URL: http://arxiv.org/abs/2105.07960v1
• Date: Mon, 17 May 2021 15:40:42 GMT
• Title: Behavior-based Neuroevolutionary Training in Reinforcement Learning
• Authors: J\"org Stork, Martin Zaefferer, Nils Eisler, Patrick Tichelmann, Thomas Bartz-Beielstein, A. E. Eiben
• Abstract summary: This work presents a hybrid algorithm that combines neuroevolutionary optimization with value-based reinforcement learning. For this purpose, we consolidate different methods to generate and optimize agent policies, creating a diverse population. Our results indicate that combining methods can enhance the sample efficiency and learning speed for evolutionary approaches.
• Score: 3.686320043830301
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In addition to their undisputed success in solving classical optimization problems, neuroevolutionary and population-based algorithms have become an alternative to standard reinforcement learning methods. However, evolutionary methods often lack the sample efficiency of standard value-based methods that leverage gathered state and value experience. If reinforcement learning for real-world problems with significant resource cost is considered, sample efficiency is essential. The enhancement of evolutionary algorithms with experience exploiting methods is thus desired and promises valuable insights. This work presents a hybrid algorithm that combines topology-changing neuroevolutionary optimization with value-based reinforcement learning. We illustrate how the behavior of policies can be used to create distance and loss functions, which benefit from stored experiences and calculated state values. They allow us to model behavior and perform a directed search in the behavior space by gradient-free evolutionary algorithms and surrogate-based optimization. For this purpose, we consolidate different methods to generate and optimize agent policies, creating a diverse population. We exemplify the performance of our algorithm on standard benchmarks and a purpose-built real-world problem. Our results indicate that combining methods can enhance the sample efficiency and learning speed for evolutionary approaches.

Related papers

• Accelerating evolutionary exploration through language model-based transfer learning [7.4439048149751095]
  We propose an approach for integrating transfer learning with gene expression programming applied to symbolic regression. The framework integrates Natural Language Processing techniques to discern correlations and recurring patterns from equations explored during previous optimizations. Our results affirm that initial solutions derived via a transfer learning mechanism enhance the algorithm's convergence rate towards improved solutions.
  arXiv  Detail & Related papers  (2024-06-07T08:05:52Z)
• Model Uncertainty in Evolutionary Optimization and Bayesian Optimization: A Comparative Analysis [5.6787965501364335]
  Black-box optimization problems are common in many real-world applications. These problems require optimization through input-output interactions without access to internal workings. Two widely used gradient-free optimization techniques are employed to address such challenges. This paper aims to elucidate the similarities and differences in the utilization of model uncertainty between these two methods.
  arXiv  Detail & Related papers  (2024-03-21T13:59:19Z)
• RLIF: Interactive Imitation Learning as Reinforcement Learning [56.997263135104504]
  We show how off-policy reinforcement learning can enable improved performance under assumptions that are similar but potentially even more practical than those of interactive imitation learning. Our proposed method uses reinforcement learning with user intervention signals themselves as rewards. This relaxes the assumption that intervening experts in interactive imitation learning should be near-optimal and enables the algorithm to learn behaviors that improve over the potential suboptimal human expert.
  arXiv  Detail & Related papers  (2023-11-21T21:05:21Z)
• Distilling Knowledge from Resource Management Algorithms to Neural Networks: A Unified Training Assistance Approach [18.841969905928337]
  knowledge distillation (KD) based algorithm distillation (AD) method is proposed in this paper to improve the performance and convergence speed of the NN-based method. This research paves the way for the integration of traditional optimization insights and emerging NN techniques in wireless communication system optimization.
  arXiv  Detail & Related papers  (2023-08-15T00:30:58Z)
• Exploring Novel Quality Diversity Methods For Generalization in Reinforcement Learning [0.0]
  The Reinforcement Learning field is strong on achievements and weak on reapplication. This paper asks whether the method of training networks improves their generalization.
  arXiv  Detail & Related papers  (2023-03-26T00:23:29Z)
• Improved Algorithms for Neural Active Learning [74.89097665112621]
  We improve the theoretical and empirical performance of neural-network(NN)-based active learning algorithms for the non-parametric streaming setting. We introduce two regret metrics by minimizing the population loss that are more suitable in active learning than the one used in state-of-the-art (SOTA) related work.
  arXiv  Detail & Related papers  (2022-10-02T05:03:38Z)
• A Boosting Approach to Reinforcement Learning [59.46285581748018]
  We study efficient algorithms for reinforcement learning in decision processes whose complexity is independent of the number of states. We give an efficient algorithm that is capable of improving the accuracy of such weak learning methods.
  arXiv  Detail & Related papers  (2021-08-22T16:00:45Z)
• DEALIO: Data-Efficient Adversarial Learning for Imitation from Observation [57.358212277226315]
  In imitation learning from observation IfO, a learning agent seeks to imitate a demonstrating agent using only observations of the demonstrated behavior without access to the control signals generated by the demonstrator. Recent methods based on adversarial imitation learning have led to state-of-the-art performance on IfO problems, but they typically suffer from high sample complexity due to a reliance on data-inefficient, model-free reinforcement learning algorithms. This issue makes them impractical to deploy in real-world settings, where gathering samples can incur high costs in terms of time, energy, and risk. We propose a more data-efficient IfO algorithm
  arXiv  Detail & Related papers  (2021-03-31T23:46:32Z)
• Evolving Reinforcement Learning Algorithms [186.62294652057062]
  We propose a method for meta-learning reinforcement learning algorithms. The learned algorithms are domain-agnostic and can generalize to new environments not seen during training. We highlight two learned algorithms which obtain good generalization performance over other classical control tasks, gridworld type tasks, and Atari games.
  arXiv  Detail & Related papers  (2021-01-08T18:55:07Z)
• Model-Augmented Actor-Critic: Backpropagating through Paths [81.86992776864729]
  Current model-based reinforcement learning approaches use the model simply as a learned black-box simulator. We show how to make more effective use of the model by exploiting its differentiability.
  arXiv  Detail & Related papers  (2020-05-16T19:18:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.