Generalized Population-Based Training for Hyperparameter Optimization in Reinforcement Learning

• URL: http://arxiv.org/abs/2404.08233v2
• Date: Tue, 23 Apr 2024 03:28:31 GMT
• Title: Generalized Population-Based Training for Hyperparameter Optimization in Reinforcement Learning
• Authors: Hui Bai, Ran Cheng,
• Abstract summary: Generalized Population-Based Training (GPBT) and Pairwise Learning (PL) PL employs a comprehensive pairwise strategy to identify performance differentials and provide holistic guidance to underperforming agents.
• Score: 10.164982368785854
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Hyperparameter optimization plays a key role in the machine learning domain. Its significance is especially pronounced in reinforcement learning (RL), where agents continuously interact with and adapt to their environments, requiring dynamic adjustments in their learning trajectories. To cater to this dynamicity, the Population-Based Training (PBT) was introduced, leveraging the collective intelligence of a population of agents learning simultaneously. However, PBT tends to favor high-performing agents, potentially neglecting the explorative potential of agents on the brink of significant advancements. To mitigate the limitations of PBT, we present the Generalized Population-Based Training (GPBT), a refined framework designed for enhanced granularity and flexibility in hyperparameter adaptation. Complementing GPBT, we further introduce Pairwise Learning (PL). Instead of merely focusing on elite agents, PL employs a comprehensive pairwise strategy to identify performance differentials and provide holistic guidance to underperforming agents. By integrating the capabilities of GPBT and PL, our approach significantly improves upon traditional PBT in terms of adaptability and computational efficiency. Rigorous empirical evaluations across a range of RL benchmarks confirm that our approach consistently outperforms not only the conventional PBT but also its Bayesian-optimized variant.

Related papers

• From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learning [62.54484062185869]
  We introduce StepAgent, which utilizes step-wise reward to optimize the agent's reinforcement learning process. We propose implicit-reward and inverse reinforcement learning techniques to facilitate agent reflection and policy adjustment.
  arXiv  Detail & Related papers  (2024-11-06T10:35:11Z)
• Hierarchical Preference Optimization: Learning to achieve goals via feasible subgoals prediction [71.81851971324187]
  This work introduces Hierarchical Preference Optimization (HPO), a novel approach to hierarchical reinforcement learning (HRL) HPO addresses non-stationarity and infeasible subgoal generation issues when solving complex robotic control tasks. Experiments on challenging robotic navigation and manipulation tasks demonstrate impressive performance of HPO, where it shows an improvement of up to 35% over the baselines.
  arXiv  Detail & Related papers  (2024-11-01T04:58:40Z)
• Simultaneous Training of First- and Second-Order Optimizers in Population-Based Reinforcement Learning [0.0]
  Population-based training (PBT) provides a method to achieve this by continuously tuning hyperparameters throughout the training. We propose an enhancement to PBT by simultaneously utilizing both first- and second-orders within a single population.
  arXiv  Detail & Related papers  (2024-08-27T21:54:26Z)
• Optimization-Derived Learning with Essential Convergence Analysis of Training and Hyper-training [52.39882976848064]
  We design a Generalized Krasnoselskii-Mann (GKM) scheme based on fixed-point iterations as our fundamental ODL module. Under the GKM scheme, a Bilevel Meta Optimization (BMO) algorithmic framework is constructed to solve the optimal training and hyper-training variables together.
  arXiv  Detail & Related papers  (2022-06-16T01:50:25Z)
• Towards Learning Universal Hyperparameter Optimizers with Transformers [57.35920571605559]
  We introduce the OptFormer, the first text-based Transformer HPO framework that provides a universal end-to-end interface for jointly learning policy and function prediction. Our experiments demonstrate that the OptFormer can imitate at least 7 different HPO algorithms, which can be further improved via its function uncertainty estimates.
  arXiv  Detail & Related papers  (2022-05-26T12:51:32Z)
• Faster Improvement Rate Population Based Training [7.661301899629696]
  This paper presents Faster Improvement Rate PBT (FIRE PBT) which addresses the problem of Population Based Training (PBT) We derive a novel fitness metric and use it to make some of the population members focus on long-term performance. Experiments show that FIRE PBT is able to outperform PBT on the ImageNet benchmark and match the performance of networks that were trained with a hand-tuned learning rate schedule.
  arXiv  Detail & Related papers  (2021-09-28T15:30:55Z)
• APS: Active Pretraining with Successor Features [96.24533716878055]
  We show that by reinterpreting and combining successorcitepHansenFast with non entropy, the intractable mutual information can be efficiently optimized. The proposed method Active Pretraining with Successor Feature (APS) explores the environment via non entropy, and the explored data can be efficiently leveraged to learn behavior.
  arXiv  Detail & Related papers  (2021-08-31T16:30:35Z)
• Dynamics Generalization via Information Bottleneck in Deep Reinforcement Learning [90.93035276307239]
  We propose an information theoretic regularization objective and an annealing-based optimization method to achieve better generalization ability in RL agents. We demonstrate the extreme generalization benefits of our approach in different domains ranging from maze navigation to robotic tasks. This work provides a principled way to improve generalization in RL by gradually removing information that is redundant for task-solving.
  arXiv  Detail & Related papers  (2020-08-03T02:24:20Z)
• Data Efficient Training for Reinforcement Learning with Adaptive Behavior Policy Sharing [29.283554268767805]
  Training deep RL model is challenging in real world applications such as production-scale health-care or recommender systems. We propose Adaptive Behavior Policy Sharing (ABPS), a data-efficient training algorithm that allows sharing of experience collected by behavior policy.
  arXiv  Detail & Related papers  (2020-02-12T20:35:31Z)
• Regularized Evolutionary Population-Based Training [11.624954122221562]
  This paper presents an algorithm called Population-Based Training (EPBT) that interleaves the training of a DNN's weights with the metalearning of loss functions. EPBT results in faster, more accurate learning on image classification benchmarks.
  arXiv  Detail & Related papers  (2020-02-11T06:28:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.