Hyperparameter Optimization Is Deceiving Us, and How to Stop It

• URL: http://arxiv.org/abs/2102.03034v1
• Date: Fri, 5 Feb 2021 07:30:43 GMT
• Title: Hyperparameter Optimization Is Deceiving Us, and How to Stop It
• Abstract summary: We show that inconsistent performance results, based on choice of hyper parameter subspace to search, are a widespread problem in ML research. We analytically characterize this problem, which we term hyper parameter deception, and show that grid search is inherently deceptive. We prove a defense with guarantees against deception, and demonstrate a defense in practice.
• Score: 38.109793771115726
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While hyperparameter optimization (HPO) is known to greatly impact learning algorithm performance, it is often treated as an empirical afterthought. Recent empirical works have highlighted the risk of this second-rate treatment of HPO. They show that inconsistent performance results, based on choice of hyperparameter subspace to search, are a widespread problem in ML research. When comparing two algorithms, J and K searching one subspace can yield the conclusion that J outperforms K, whereas searching another can entail the opposite result. In short, your choice of hyperparameters can deceive you. We provide a theoretical complement to this prior work: We analytically characterize this problem, which we term hyperparameter deception, and show that grid search is inherently deceptive. We prove a defense with guarantees against deception, and demonstrate a defense in practice.

Related papers

• On the importance of hyperparameters in initializing parameterized quantum circuits [1.6206092886889898]
  In this paper, we focus on the problem of finding performant initial parameters for a given Quantum Circuits (PQC)<n>Our empirical results indicate that our algorithm consistently leads to selection of performant initial parameters tuned specifically to the ansatz.<n>Our algorithm does not emphnegatively affect the barren plateau phenomenon.
  arXiv  Detail & Related papers  (2026-04-23T04:21:11Z)
• POCAII: Parameter Optimization with Conscious Allocation using Iterative Intelligence [4.478575931884855]
  POCAII is a flexible scheme for managing a hyper parameter optimization budget.<n>It shows superior performance in low-budget hyper parameter optimization regimes.<n>It has wide applications to real-world problems.
  arXiv  Detail & Related papers  (2025-05-16T23:05:07Z)
• HyperArm Bandit Optimization: A Novel approach to Hyperparameter Optimization and an Analysis of Bandit Algorithms in Stochastic and Adversarial Settings [0.0]
  The paper introduces a novel framework, HyperArm Bandit Optimization (HABO), which applies EXP3 to hyper parameter tuning in machine learning models. HABO treats individual hyper parameters as super-arms, and its potential configurations as sub-arms, enabling dynamic resource allocation and efficient exploration.
  arXiv  Detail & Related papers  (2025-03-13T11:50:28Z)
• Hyperparameter Optimization Can Even be Harmful in Off-Policy Learning and How to Deal with It [20.312864152544954]
  We show that naively applying an unbiased estimator of the generalization performance as a surrogate objective in HPO can cause an unexpected failure. We propose simple and computationally efficient corrections to the typical HPO procedure to deal with the aforementioned issues simultaneously.
  arXiv  Detail & Related papers  (2024-04-23T14:34:16Z)
• PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learning [49.92394599459274]
  We propose PriorBand, an HPO algorithm tailored to Deep Learning (DL) pipelines. We show its robustness across a range of DL benchmarks and show its gains under informative expert input and against poor expert beliefs.
  arXiv  Detail & Related papers  (2023-06-21T16:26:14Z)
• HyperTime: Hyperparameter Optimization for Combating Temporal Distribution Shifts [26.205660967039087]
  We use the lexicographic priority order on average validation loss and worst-case validation loss over chronological validation sets. We show the strong empirical performance of the proposed method on multiple machine learning tasks with temporal distribution shifts.
  arXiv  Detail & Related papers  (2023-05-28T19:41:23Z)
• Online Continuous Hyperparameter Optimization for Generalized Linear Contextual Bandits [55.03293214439741]
  In contextual bandits, an agent sequentially makes actions from a time-dependent action set based on past experience. We propose the first online continuous hyperparameter tuning framework for contextual bandits. We show that it could achieve a sublinear regret in theory and performs consistently better than all existing methods on both synthetic and real datasets.
  arXiv  Detail & Related papers  (2023-02-18T23:31:20Z)
• HyperNP: Interactive Visual Exploration of Multidimensional Projection Hyperparameters [61.354362652006834]
  HyperNP is a scalable method that allows for real-time interactive exploration of projection methods by training neural network approximations. We evaluate the performance of the HyperNP across three datasets in terms of performance and speed.
  arXiv  Detail & Related papers  (2021-06-25T17:28:14Z)
• An Empirical Study on Hyperparameter Optimization for Fine-Tuning Pre-trained Language Models [10.599060942412144]
  We study and report three HPO algorithms' performances on fine-tuning two state-of-the-art language models on the GLUE dataset. We find that using the same time budget, HPO often fails to outperform grid search due to two reasons: insufficient time budget and overfitting.
  arXiv  Detail & Related papers  (2021-06-17T01:58:32Z)
• Optimizing Large-Scale Hyperparameters via Automated Learning Algorithm [97.66038345864095]
  We propose a new hyperparameter optimization method with zeroth-order hyper-gradients (HOZOG) Specifically, we first formulate hyperparameter optimization as an A-based constrained optimization problem. Then, we use the average zeroth-order hyper-gradients to update hyper parameters.
  arXiv  Detail & Related papers  (2021-02-17T21:03:05Z)
• Quantity vs. Quality: On Hyperparameter Optimization for Deep Reinforcement Learning [7.559006677497745]
  Reinforcement learning algorithms can show strong variation in performance between training runs with different random seeds. We benchmark whether it is better to explore a large quantity of hyperparameter settings via pruning of bad performers, or if it is better to aim for quality of collected results by using repetitions.
  arXiv  Detail & Related papers  (2020-07-29T05:12:34Z)
• An Asymptotically Optimal Multi-Armed Bandit Algorithm and Hyperparameter Optimization [48.5614138038673]
  We propose an efficient and robust bandit-based algorithm called Sub-Sampling (SS) in the scenario of hyper parameter search evaluation. We also develop a novel hyper parameter optimization algorithm called BOSS. Empirical studies validate our theoretical arguments of SS and demonstrate the superior performance of BOSS on a number of applications.
  arXiv  Detail & Related papers  (2020-07-11T03:15:21Z)
• Weighted Random Search for Hyperparameter Optimization [0.0]
  We introduce an improved version of Random Search (RS), used here for hyper parameter optimization of machine learning algorithms. We generate new values for each hyper parameter with a probability of change, unlike the standard RS. Within the same computational budget, our method yields better results than the standard RS.
  arXiv  Detail & Related papers  (2020-04-03T15:41:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.