AutoRL Hyperparameter Landscapes

• URL: http://arxiv.org/abs/2304.02396v4
• Date: Mon, 5 Jun 2023 06:00:17 GMT
• Title: AutoRL Hyperparameter Landscapes
• Authors: Aditya Mohan, Carolin Benjamins, Konrad Wienecke, Alexander Dockhorn, Marius Lindauer
• Abstract summary: Reinforcement Learning (RL) has shown to be capable of producing impressive results, but its use is limited by the impact of its hyperparameters on performance. We propose an approach to build and analyze these hyperparameter landscapes not just for one point in time but at multiple points in time throughout training. This supports the theory that hyperparameters should be dynamically adjusted during training and shows the potential for more insights on AutoRL problems that can be gained through landscape analyses.
• Score: 69.15927869840918
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Although Reinforcement Learning (RL) has shown to be capable of producing impressive results, its use is limited by the impact of its hyperparameters on performance. This often makes it difficult to achieve good results in practice. Automated RL (AutoRL) addresses this difficulty, yet little is known about the dynamics of the hyperparameter landscapes that hyperparameter optimization (HPO) methods traverse in search of optimal configurations. In view of existing AutoRL approaches dynamically adjusting hyperparameter configurations, we propose an approach to build and analyze these hyperparameter landscapes not just for one point in time but at multiple points in time throughout training. Addressing an important open question on the legitimacy of such dynamic AutoRL approaches, we provide thorough empirical evidence that the hyperparameter landscapes strongly vary over time across representative algorithms from RL literature (DQN, PPO, and SAC) in different kinds of environments (Cartpole, Bipedal Walker, and Hopper) This supports the theory that hyperparameters should be dynamically adjusted during training and shows the potential for more insights on AutoRL problems that can be gained through landscape analyses. Our code can be found at https://github.com/automl/AutoRL-Landscape

Related papers

• ARLBench: Flexible and Efficient Benchmarking for Hyperparameter Optimization in Reinforcement Learning [42.33815055388433]
  ARLBench is a benchmark for hyperparameter optimization (HPO) in reinforcement learning (RL) It allows comparisons of diverse HPO approaches while being highly efficient in evaluation. ARLBench is an efficient, flexible, and future-oriented foundation for research on AutoRL.
  arXiv  Detail & Related papers  (2024-09-27T15:22:28Z)
• A Framework for History-Aware Hyperparameter Optimisation in Reinforcement Learning [8.659973888018781]
  A Reinforcement Learning (RL) system depends on a set of initial conditions that affect the system's performance. We propose a framework based on integrating complex event processing and temporal models, to alleviate these trade-offs. We tested the proposed approach in a 5G mobile communications case study that uses DQN, a variant of RL, for its decision-making.
  arXiv  Detail & Related papers  (2023-03-09T11:30:40Z)
• Hyperparameters in Contextual RL are Highly Situational [16.328866317851183]
  Reinforcement Learning (RL) has shown impressive results in games and simulation, but real-world application suffers from its instability under changing environment conditions. We show that the hyper parameters found by HPO methods are not only dependent on the problem at hand, but even on how well the state describes the environment dynamics.
  arXiv  Detail & Related papers  (2022-12-21T09:38:18Z)
• Hyper-Parameter Auto-Tuning for Sparse Bayesian Learning [72.83293818245978]
  We design and learn a neural network (NN)-based auto-tuner for hyper- parameter tuning in sparse Bayesian learning. We show that considerable improvement in convergence rate and recovery performance can be achieved.
  arXiv  Detail & Related papers  (2022-11-09T12:34:59Z)
• Automating DBSCAN via Deep Reinforcement Learning [73.82740568765279]
  We propose a novel Deep Reinforcement Learning guided automatic DBSCAN parameters search framework, namely DRL-DBSCAN. The framework models the process of adjusting the parameter search direction by perceiving the clustering environment as a Markov decision process. The framework consistently improves DBSCAN clustering accuracy by up to 26% and 25% respectively.
  arXiv  Detail & Related papers  (2022-08-09T04:40:11Z)
• AUTOMATA: Gradient Based Data Subset Selection for Compute-Efficient Hyper-parameter Tuning [72.54359545547904]
  We propose a gradient-based subset selection framework for hyper- parameter tuning. We show that using gradient-based data subsets for hyper- parameter tuning achieves significantly faster turnaround times and speedups of 3$times$-30$times$.
  arXiv  Detail & Related papers  (2022-03-15T19:25:01Z)
• On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learning [27.36718899899319]
  Model-based Reinforcement Learning (MBRL) is a promising framework for learning control in a data-efficient manner. MBRL typically requires significant human expertise before it can be applied to new problems and domains.
  arXiv  Detail & Related papers  (2021-02-26T18:57:47Z)
• Online hyperparameter optimization by real-time recurrent learning [57.01871583756586]
  Our framework takes advantage of the analogy between hyperparameter optimization and parameter learning in neural networks (RNNs) It adapts a well-studied family of online learning algorithms for RNNs to tune hyperparameters and network parameters simultaneously. This procedure yields systematically better generalization performance compared to standard methods, at a fraction of wallclock time.
  arXiv  Detail & Related papers  (2021-02-15T19:36:18Z)
• Sample-Efficient Automated Deep Reinforcement Learning [33.53903358611521]
  We propose a population-based automated RL framework to meta-optimize arbitrary off-policy RL algorithms. By sharing the collected experience across the population, we substantially increase the sample efficiency of the meta-optimization. We demonstrate the capabilities of our sample-efficient AutoRL approach in a case study with the popular TD3 algorithm in the MuJoCo benchmark suite.
  arXiv  Detail & Related papers  (2020-09-03T10:04:06Z)
• Hyperparameter Selection for Offline Reinforcement Learning [61.92834684647419]
  offline reinforcement learning (RL purely from logged data) is an important avenue for deploying RL techniques in real-world scenarios. Existing hyperparameter selection methods for offline RL break the offline assumption.
  arXiv  Detail & Related papers  (2020-07-17T15:30:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.