Derivative-Free Reinforcement Learning: A Review

• URL: http://arxiv.org/abs/2102.05710v1
• Date: Wed, 10 Feb 2021 19:29:22 GMT
• Title: Derivative-Free Reinforcement Learning: A Review
• Authors: Hong Qian and Yang Yu
• Abstract summary: Reinforcement learning is about learning agent models that make the best sequential decisions in unknown environments. Derivative-free optimization, meanwhile, is capable of solving sophisticated problems. This article summarizes methods of derivative-free reinforcement learning to date, and organizes the methods in aspects including parameter updating, model selection, exploration, and parallel/distributed methods.
• Score: 11.568151821073952
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement learning is about learning agent models that make the best sequential decisions in unknown environments. In an unknown environment, the agent needs to explore the environment while exploiting the collected information, which usually forms a sophisticated problem to solve. Derivative-free optimization, meanwhile, is capable of solving sophisticated problems. It commonly uses a sampling-and-updating framework to iteratively improve the solution, where exploration and exploitation are also needed to be well balanced. Therefore, derivative-free optimization deals with a similar core issue as reinforcement learning, and has been introduced in reinforcement learning approaches, under the names of learning classifier systems and neuroevolution/evolutionary reinforcement learning. Although such methods have been developed for decades, recently, derivative-free reinforcement learning exhibits attracting increasing attention. However, recent survey on this topic is still lacking. In this article, we summarize methods of derivative-free reinforcement learning to date, and organize the methods in aspects including parameter updating, model selection, exploration, and parallel/distributed methods. Moreover, we discuss some current limitations and possible future directions, hoping that this article could bring more attentions to this topic and serve as a catalyst for developing novel and efficient approaches.

Related papers

• A Unified Framework for Neural Computation and Learning Over Time [56.44910327178975]
  Hamiltonian Learning is a novel unified framework for learning with neural networks "over time" It is based on differential equations that: (i) can be integrated without the need of external software solvers; (ii) generalize the well-established notion of gradient-based learning in feed-forward and recurrent networks; (iii) open to novel perspectives.
  arXiv  Detail & Related papers  (2024-09-18T14:57:13Z)
• Learning in Hybrid Active Inference Models [0.8749675983608172]
  We present a novel hierarchical hybrid active inference agent in which a high-level discrete active inference planner sits above a low-level continuous active inference controller. We make use of recent work in recurrent switching linear dynamical systems which implement end-to-end learning of meaningful discrete representations. We apply our model to the sparse Continuous Mountain Car task, demonstrating fast system identification via enhanced exploration and successful planning.
  arXiv  Detail & Related papers  (2024-09-02T08:41:45Z)
• A Unified and General Framework for Continual Learning [58.72671755989431]
  Continual Learning (CL) focuses on learning from dynamic and changing data distributions while retaining previously acquired knowledge. Various methods have been developed to address the challenge of catastrophic forgetting, including regularization-based, Bayesian-based, and memory-replay-based techniques. This research aims to bridge this gap by introducing a comprehensive and overarching framework that encompasses and reconciles these existing methodologies.
  arXiv  Detail & Related papers  (2024-03-20T02:21:44Z)
• Model-Based Reinforcement Learning Control of Reaction-Diffusion Problems [0.0]
  reinforcement learning has been applied to decision-making in several applications, most notably in games. We introduce two novel reward functions to drive the flow of the transported field. Results show that certain controls can be implemented successfully in these applications.
  arXiv  Detail & Related papers  (2024-02-22T11:06:07Z)
• RLIF: Interactive Imitation Learning as Reinforcement Learning [56.997263135104504]
  We show how off-policy reinforcement learning can enable improved performance under assumptions that are similar but potentially even more practical than those of interactive imitation learning. Our proposed method uses reinforcement learning with user intervention signals themselves as rewards. This relaxes the assumption that intervening experts in interactive imitation learning should be near-optimal and enables the algorithm to learn behaviors that improve over the potential suboptimal human expert.
  arXiv  Detail & Related papers  (2023-11-21T21:05:21Z)
• Reusable Options through Gradient-based Meta Learning [24.59017394648942]
  Several deep learning approaches were proposed to learn temporal abstractions in the form of options in an end-to-end manner. We frame the problem of learning options as a gradient-based meta-learning problem. We show that our method is able to learn transferable components which accelerate learning and performs better than existing prior methods.
  arXiv  Detail & Related papers  (2022-12-22T14:19:35Z)
• Mixture-of-Variational-Experts for Continual Learning [0.0]
  We propose an optimality principle that facilitates a trade-off between learning and forgetting. We propose a neural network layer for continual learning, called Mixture-of-Variational-Experts (MoVE) Our experiments on variants of the MNIST and CIFAR10 datasets demonstrate the competitive performance of MoVE layers.
  arXiv  Detail & Related papers  (2021-10-25T06:32:06Z)
• Transfer Learning in Deep Reinforcement Learning: A Survey [64.36174156782333]
  Reinforcement learning is a learning paradigm for solving sequential decision-making problems. Recent years have witnessed remarkable progress in reinforcement learning upon the fast development of deep neural networks. transfer learning has arisen to tackle various challenges faced by reinforcement learning.
  arXiv  Detail & Related papers  (2020-09-16T18:38:54Z)
• Importance Weighted Policy Learning and Adaptation [89.46467771037054]
  We study a complementary approach which is conceptually simple, general, modular and built on top of recent improvements in off-policy learning. The framework is inspired by ideas from the probabilistic inference literature and combines robust off-policy learning with a behavior prior. Our approach achieves competitive adaptation performance on hold-out tasks compared to meta reinforcement learning baselines and can scale to complex sparse-reward scenarios.
  arXiv  Detail & Related papers  (2020-09-10T14:16:58Z)
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems [108.81683598693539]
  offline reinforcement learning algorithms hold tremendous promise for making it possible to turn large datasets into powerful decision making engines. We will aim to provide the reader with an understanding of these challenges, particularly in the context of modern deep reinforcement learning methods.
  arXiv  Detail & Related papers  (2020-05-04T17:00:15Z)
• A Survey of Reinforcement Learning Techniques: Strategies, Recent Development, and Future Directions [0.0]
  Reinforcement learning influences the system to take actions within an arbitrary environment. This paper focuses on various dimensions including challenges, the recent development of different state-of-the-art techniques, and future directions.
  arXiv  Detail & Related papers  (2020-01-19T23:51:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.