A Method for Evaluating Hyperparameter Sensitivity in Reinforcement Learning

• URL: http://arxiv.org/abs/2412.07165v2
• Date: Tue, 04 Feb 2025 05:17:26 GMT
• Title: A Method for Evaluating Hyperparameter Sensitivity in Reinforcement Learning
• Authors: Jacob Adkins, Michael Bowling, Adam White,
• Abstract summary: This work proposes a new empirical methodology for studying, comparing, and quantifying the sensitivity of an algorithm's performance to hyperparameter tuning. The results suggest that several algorithmic performance improvements may, in fact, be a result of an increased reliance on hyperparameter tuning.
• Score: 11.929813643723413
• License:
• Abstract: The performance of modern reinforcement learning algorithms critically relies on tuning ever-increasing numbers of hyperparameters. Often, small changes in a hyperparameter can lead to drastic changes in performance, and different environments require very different hyperparameter settings to achieve state-of-the-art performance reported in the literature. We currently lack a scalable and widely accepted approach to characterizing these complex interactions. This work proposes a new empirical methodology for studying, comparing, and quantifying the sensitivity of an algorithm's performance to hyperparameter tuning for a given set of environments. We then demonstrate the utility of this methodology by assessing the hyperparameter sensitivity of several commonly used normalization variants of PPO. The results suggest that several algorithmic performance improvements may, in fact, be a result of an increased reliance on hyperparameter tuning.

Related papers

• Sample complexity of data-driven tuning of model hyperparameters in neural networks with structured parameter-dependent dual function [24.457000214575245]
  We introduce a new technique to characterize the discontinuities and oscillations of the utility function on any fixed problem instance. This can be used to show that the learning theoretic complexity of the corresponding family of utility functions is bounded.
  arXiv  Detail & Related papers  (2025-01-23T15:10:51Z)
• Beyond algorithm hyperparameters: on preprocessing hyperparameters and associated pitfalls in machine learning applications [0.30786914102688595]
  This paper reviews and empirically illustrates different procedures for generating and evaluating prediction models. By highlighting potential pitfalls, especially those that may lead to exaggerated performance claims, this review aims to further improve the quality of predictive modeling in ML applications.
  arXiv  Detail & Related papers  (2024-12-04T17:29:10Z)
• On the consistency of hyper-parameter selection in value-based deep reinforcement learning [13.133865673667394]
  This paper conducts an empirical study focusing on the reliability of hyper- parameter selection for value-based deep reinforcement learning agents. Our findings help establish which hyper- parameters are most critical to tune, and help clarify which tunings remain consistent across different training regimes.
  arXiv  Detail & Related papers  (2024-06-25T13:06:09Z)
• ETHER: Efficient Finetuning of Large-Scale Models with Hyperplane Reflections [59.839926875976225]
  We propose the ETHER transformation family, which performs Efficient fineTuning via HypErplane Reflections. In particular, we introduce ETHER and its relaxation ETHER+, which match or outperform existing PEFT methods with significantly fewer parameters.
  arXiv  Detail & Related papers  (2024-05-30T17:26:02Z)
• Hyperparameter Adaptive Search for Surrogate Optimization: A Self-Adjusting Approach [1.6317061277457001]
  Surrogate optimization (SO) algorithms have shown promise for optimizing expensive black-box functions. Our approach identifies and modifies the most influential hyper parameters specific to each problem and SO approach. Experimental results demonstrate the effectiveness of HASSO in enhancing the performance of various SO algorithms.
  arXiv  Detail & Related papers  (2023-10-12T01:26:05Z)
• Multi-objective hyperparameter optimization with performance uncertainty [62.997667081978825]
  This paper presents results on multi-objective hyperparameter optimization with uncertainty on the evaluation of Machine Learning algorithms. We combine the sampling strategy of Tree-structured Parzen Estimators (TPE) with the metamodel obtained after training a Gaussian Process Regression (GPR) with heterogeneous noise. Experimental results on three analytical test functions and three ML problems show the improvement over multi-objective TPE and GPR.
  arXiv  Detail & Related papers  (2022-09-09T14:58:43Z)
• Towards Learning Universal Hyperparameter Optimizers with Transformers [57.35920571605559]
  We introduce the OptFormer, the first text-based Transformer HPO framework that provides a universal end-to-end interface for jointly learning policy and function prediction. Our experiments demonstrate that the OptFormer can imitate at least 7 different HPO algorithms, which can be further improved via its function uncertainty estimates.
  arXiv  Detail & Related papers  (2022-05-26T12:51:32Z)
• AUTOMATA: Gradient Based Data Subset Selection for Compute-Efficient Hyper-parameter Tuning [72.54359545547904]
  We propose a gradient-based subset selection framework for hyper- parameter tuning. We show that using gradient-based data subsets for hyper- parameter tuning achieves significantly faster turnaround times and speedups of 3$times$-30$times$.
  arXiv  Detail & Related papers  (2022-03-15T19:25:01Z)
• Online hyperparameter optimization by real-time recurrent learning [57.01871583756586]
  Our framework takes advantage of the analogy between hyperparameter optimization and parameter learning in neural networks (RNNs) It adapts a well-studied family of online learning algorithms for RNNs to tune hyperparameters and network parameters simultaneously. This procedure yields systematically better generalization performance compared to standard methods, at a fraction of wallclock time.
  arXiv  Detail & Related papers  (2021-02-15T19:36:18Z)
• Efficient Hyperparameter Tuning with Dynamic Accuracy Derivative-Free Optimization [0.27074235008521236]
  We apply a recent dynamic accuracy derivative-free optimization method to hyperparameter tuning. This method allows inexact evaluations of the learning problem while retaining convergence guarantees. We demonstrate its robustness and efficiency compared to a fixed accuracy approach.
  arXiv  Detail & Related papers  (2020-11-06T00:59:51Z)
• Adaptive Gradient Method with Resilience and Momentum [120.83046824742455]
  We propose an Adaptive Gradient Method with Resilience and Momentum (AdaRem) AdaRem adjusts the parameter-wise learning rate according to whether the direction of one parameter changes in the past is aligned with the direction of the current gradient. Our method outperforms previous adaptive learning rate-based algorithms in terms of the training speed and the test error.
  arXiv  Detail & Related papers  (2020-10-21T14:49:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.