Hyperparameter Optimization in Machine Learning

• URL: http://arxiv.org/abs/2410.22854v1
• Date: Wed, 30 Oct 2024 09:39:22 GMT
• Title: Hyperparameter Optimization in Machine Learning
• Authors: Luca Franceschi, Michele Donini, Valerio Perrone, Aaron Klein, Cédric Archambeau, Matthias Seeger, Massimiliano Pontil, Paolo Frasconi,
• Abstract summary: Hyperparameters are configuration variables controlling the behavior of machine learning algorithms. The choice of their values determine the effectiveness of systems based on these technologies. We present a unified treatment of hyperparameter optimization, providing the reader with examples and insights into the state-of-the-art.
• Score: 34.356747514732966
• License:
• Abstract: Hyperparameters are configuration variables controlling the behavior of machine learning algorithms. They are ubiquitous in machine learning and artificial intelligence and the choice of their values determine the effectiveness of systems based on these technologies. Manual hyperparameter search is often unsatisfactory and becomes unfeasible when the number of hyperparameters is large. Automating the search is an important step towards automating machine learning, freeing researchers and practitioners alike from the burden of finding a good set of hyperparameters by trial and error. In this survey, we present a unified treatment of hyperparameter optimization, providing the reader with examples and insights into the state-of-the-art. We cover the main families of techniques to automate hyperparameter search, often referred to as hyperparameter optimization or tuning, including random and quasi-random search, bandit-, model- and gradient- based approaches. We further discuss extensions, including online, constrained, and multi-objective formulations, touch upon connections with other fields such as meta-learning and neural architecture search, and conclude with open questions and future research directions.

Related papers

• A Linear Programming Enhanced Genetic Algorithm for Hyperparameter Tuning in Machine Learning [0.34530027457862006]
  In this paper, we formulate the hyperparameter tuning problem in machine learning as a bilevel program. The bilevel program is solved using a micro genetic algorithm that is enhanced with a linear program. We test the performance of the proposed approach on two datasets, MNIST and CIFAR-10.
  arXiv  Detail & Related papers  (2024-06-30T07:11:00Z)
• Scrap Your Schedules with PopDescent [0.0]
  Population Descent (PopDescent) is a memetic, population-based search technique. We show that PopDescent converges faster than existing search methods, finding model parameters with test-loss values up to 18% lower. Our trials on standard machine learning vision tasks show that PopDescent converges faster than existing search methods, finding model parameters with test-loss values up to 18% lower.
  arXiv  Detail & Related papers  (2023-10-23T08:11:17Z)
• AutoRL Hyperparameter Landscapes [69.15927869840918]
  Reinforcement Learning (RL) has shown to be capable of producing impressive results, but its use is limited by the impact of its hyperparameters on performance. We propose an approach to build and analyze these hyperparameter landscapes not just for one point in time but at multiple points in time throughout training. This supports the theory that hyperparameters should be dynamically adjusted during training and shows the potential for more insights on AutoRL problems that can be gained through landscape analyses.
  arXiv  Detail & Related papers  (2023-04-05T12:14:41Z)
• AUTOMATA: Gradient Based Data Subset Selection for Compute-Efficient Hyper-parameter Tuning [72.54359545547904]
  We propose a gradient-based subset selection framework for hyper- parameter tuning. We show that using gradient-based data subsets for hyper- parameter tuning achieves significantly faster turnaround times and speedups of 3$times$-30$times$.
  arXiv  Detail & Related papers  (2022-03-15T19:25:01Z)
• To tune or not to tune? An Approach for Recommending Important Hyperparameters [2.121963121603413]
  We consider building the relationship between the performance of the machine learning models and their hyperparameters to discover the trend and gain insights. Our results enable users to decide whether it is worth conducting a possibly time-consuming tuning strategy.
  arXiv  Detail & Related papers  (2021-08-30T08:54:58Z)
• HyperNP: Interactive Visual Exploration of Multidimensional Projection Hyperparameters [61.354362652006834]
  HyperNP is a scalable method that allows for real-time interactive exploration of projection methods by training neural network approximations. We evaluate the performance of the HyperNP across three datasets in terms of performance and speed.
  arXiv  Detail & Related papers  (2021-06-25T17:28:14Z)
• Search Algorithms for Automated Hyper-Parameter Tuning [1.2233362977312945]
  We develop two automated Hyper- Optimization methods, namely grid search and random search, to assess and improve a previous study's performance. Experiment results show that applying random search and grid search on machine learning algorithms improves accuracy.
  arXiv  Detail & Related papers  (2021-04-29T22:11:52Z)
• Optimizing Large-Scale Hyperparameters via Automated Learning Algorithm [97.66038345864095]
  We propose a new hyperparameter optimization method with zeroth-order hyper-gradients (HOZOG) Specifically, we first formulate hyperparameter optimization as an A-based constrained optimization problem. Then, we use the average zeroth-order hyper-gradients to update hyper parameters.
  arXiv  Detail & Related papers  (2021-02-17T21:03:05Z)
• Online hyperparameter optimization by real-time recurrent learning [57.01871583756586]
  Our framework takes advantage of the analogy between hyperparameter optimization and parameter learning in neural networks (RNNs) It adapts a well-studied family of online learning algorithms for RNNs to tune hyperparameters and network parameters simultaneously. This procedure yields systematically better generalization performance compared to standard methods, at a fraction of wallclock time.
  arXiv  Detail & Related papers  (2021-02-15T19:36:18Z)
• On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice [10.350337750192997]
  We introduce several state-of-the-art optimization techniques and discuss how to apply them to machine learning algorithms. This paper will help industrial users, data analysts, and researchers to better develop machine learning models.
  arXiv  Detail & Related papers  (2020-07-30T21:11:01Z)
• An Asymptotically Optimal Multi-Armed Bandit Algorithm and Hyperparameter Optimization [48.5614138038673]
  We propose an efficient and robust bandit-based algorithm called Sub-Sampling (SS) in the scenario of hyper parameter search evaluation. We also develop a novel hyper parameter optimization algorithm called BOSS. Empirical studies validate our theoretical arguments of SS and demonstrate the superior performance of BOSS on a number of applications.
  arXiv  Detail & Related papers  (2020-07-11T03:15:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.