Automated Reinforcement Learning (AutoRL): A Survey and Open Problems

• URL: http://arxiv.org/abs/2201.03916v1
• Date: Tue, 11 Jan 2022 12:41:43 GMT
• Title: Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Authors: Jack Parker-Holder, Raghu Rajan, Xingyou Song, Andr\'e Biedenkapp, Yingjie Miao, Theresa Eimer, Baohe Zhang, Vu Nguyen, Roberto Calandra, Aleksandra Faust, Frank Hutter, Marius Lindauer
• Abstract summary: Automated Reinforcement Learning (AutoRL) involves not only standard applications of AutoML but also includes additional challenges unique to RL. We provide a common taxonomy, discuss each area in detail and pose open problems which would be of interest to researchers going forward.
• Score: 92.73407630874841
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The combination of Reinforcement Learning (RL) with deep learning has led to a series of impressive feats, with many believing (deep) RL provides a path towards generally capable agents. However, the success of RL agents is often highly sensitive to design choices in the training process, which may require tedious and error-prone manual tuning. This makes it challenging to use RL for new problems, while also limits its full potential. In many other areas of machine learning, AutoML has shown it is possible to automate such design choices and has also yielded promising initial results when applied to RL. However, Automated Reinforcement Learning (AutoRL) involves not only standard applications of AutoML but also includes additional challenges unique to RL, that naturally produce a different set of methods. As such, AutoRL has been emerging as an important area of research in RL, providing promise in a variety of applications from RNA design to playing games such as Go. Given the diversity of methods and environments considered in RL, much of the research has been conducted in distinct subfields, ranging from meta-learning to evolution. In this survey we seek to unify the field of AutoRL, we provide a common taxonomy, discuss each area in detail and pose open problems which would be of interest to researchers going forward.

Related papers

• A Benchmark Environment for Offline Reinforcement Learning in Racing Games [54.83171948184851]
  Offline Reinforcement Learning (ORL) is a promising approach to reduce the high sample complexity of traditional Reinforcement Learning (RL) This paper introduces OfflineMania, a novel environment for ORL research. It is inspired by the iconic TrackMania series and developed using the Unity 3D game engine.
  arXiv  Detail & Related papers  (2024-07-12T16:44:03Z)
• Adaptive $Q$-Network: On-the-fly Target Selection for Deep Reinforcement Learning [18.579378919155864]
  We propose Adaptive $Q$Network (AdaQN) to take into account the non-stationarity of the optimization procedure without requiring additional samples. AdaQN is theoretically sound and empirically validate it in MuJoCo control problems and Atari $2600 games.
  arXiv  Detail & Related papers  (2024-05-25T11:57:43Z)
• ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL [80.10358123795946]
  We develop a framework for building multi-turn RL algorithms for fine-tuning large language models. Our framework adopts a hierarchical RL approach and runs two RL algorithms in parallel. Empirically, we find that ArCHer significantly improves efficiency and performance on agent tasks.
  arXiv  Detail & Related papers  (2024-02-29T18:45:56Z)
• A Survey of Meta-Reinforcement Learning [69.76165430793571]
  We cast the development of better RL algorithms as a machine learning problem itself in a process called meta-RL. We discuss how, at a high level, meta-RL research can be clustered based on the presence of a task distribution and the learning budget available for each individual task. We conclude by presenting the open problems on the path to making meta-RL part of the standard toolbox for a deep RL practitioner.
  arXiv  Detail & Related papers  (2023-01-19T12:01:41Z)
• Understanding the Complexity Gains of Single-Task RL with a Curriculum [83.46923851724408]
  Reinforcement learning (RL) problems can be challenging without well-shaped rewards. We provide a theoretical framework that reformulates a single-task RL problem as a multi-task RL problem defined by a curriculum. We show that sequentially solving each task in the multi-task RL problem is more computationally efficient than solving the original single-task problem.
  arXiv  Detail & Related papers  (2022-12-24T19:46:47Z)
• Pretraining in Deep Reinforcement Learning: A Survey [17.38360092869849]
  Pretraining has shown to be effective in acquiring transferable knowledge. Due to the nature of reinforcement learning, pretraining in this field is faced with unique challenges.
  arXiv  Detail & Related papers  (2022-11-08T02:17:54Z)
• Automated Reinforcement Learning: An Overview [6.654552816487819]
  Reinforcement Learning and Deep Reinforcement Learning are popular methods for solving sequential decision making problems. In this article, we explore the literature and present recent work that can be used in automated RL.
  arXiv  Detail & Related papers  (2022-01-13T14:28:06Z)
• Formula RL: Deep Reinforcement Learning for Autonomous Racing using Telemetry Data [4.042350304426975]
  We frame the problem as a reinforcement learning task with a multidimensional input consisting of the vehicle telemetry, and a continuous action space. We put 10 variants of deep deterministic policy gradient (DDPG) to race in two experiments. Our studies show that models trained with RL are not only able to drive faster than the baseline open source handcrafted bots but also generalize to unknown tracks.
  arXiv  Detail & Related papers  (2021-04-22T14:40:12Z)
• Investigating Value of Curriculum Reinforcement Learning in Autonomous Driving Under Diverse Road and Weather Conditions [0.0]
  This paper focuses on investigating the value of curriculum reinforcement learning in autonomous driving applications. We setup several different driving scenarios in a realistic driving simulator, with varying road complexity and weather conditions. Results show that curriculum RL can yield significant gains in complex driving tasks, both in terms of driving performance and sample complexity.
  arXiv  Detail & Related papers  (2021-03-14T12:05:05Z)
• How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned [111.06812202454364]
  We present a number of case studies involving robotic deep RL. We discuss commonly perceived challenges in deep RL and how they have been addressed in these works. We also provide an overview of other outstanding challenges, many of which are unique to the real-world robotics setting.
  arXiv  Detail & Related papers  (2021-02-04T22:09:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.