A Comparative Study of Hyperparameter Tuning Methods

• URL: http://arxiv.org/abs/2408.16425v1
• Date: Thu, 29 Aug 2024 10:35:07 GMT
• Title: A Comparative Study of Hyperparameter Tuning Methods
• Authors: Subhasis Dasgupta, Jaydip Sen,
• Abstract summary: Tree-structured Parzen Estimator (TPE), Genetic Search, and Random Search are evaluated across regression and classification tasks. Random Search excelled in regression tasks, while TPE was more effective for classification tasks.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The study emphasizes the challenge of finding the optimal trade-off between bias and variance, especially as hyperparameter optimization increases in complexity. Through empirical analysis, three hyperparameter tuning algorithms Tree-structured Parzen Estimator (TPE), Genetic Search, and Random Search are evaluated across regression and classification tasks. The results show that nonlinear models, with properly tuned hyperparameters, significantly outperform linear models. Interestingly, Random Search excelled in regression tasks, while TPE was more effective for classification tasks. This suggests that there is no one-size-fits-all solution, as different algorithms perform better depending on the task and model type. The findings underscore the importance of selecting the appropriate tuning method and highlight the computational challenges involved in optimizing machine learning models, particularly as search spaces expand.

Related papers

• End-to-End Learning for Fair Multiobjective Optimization Under Uncertainty [55.04219793298687]
  The Predict-Then-Forecast (PtO) paradigm in machine learning aims to maximize downstream decision quality. This paper extends the PtO methodology to optimization problems with nondifferentiable Ordered Weighted Averaging (OWA) objectives. It shows how optimization of OWA functions can be effectively integrated with parametric prediction for fair and robust optimization under uncertainty.
  arXiv  Detail & Related papers  (2024-02-12T16:33:35Z)
• Agent-based Collaborative Random Search for Hyper-parameter Tuning and Global Function Optimization [0.0]
  This paper proposes an agent-based collaborative technique for finding near-optimal values for any arbitrary set of hyper- parameters in a machine learning model. The behavior of the presented model, specifically against the changes in its design parameters, is investigated in both machine learning and global function optimization applications.
  arXiv  Detail & Related papers  (2023-03-03T21:10:17Z)
• Enhancing Machine Learning Model Performance with Hyper Parameter Optimization: A Comparative Study [0.0]
  One of the most critical issues in machine learning is the selection of appropriate hyper parameters for training models. Hyper parameter optimization (HPO) is a popular topic that artificial intelligence studies have focused on recently. In this study, classical methods, such as grid, random search and Bayesian optimization, and population-based algorithms, such as genetic algorithms and particle swarm optimization, are discussed.
  arXiv  Detail & Related papers  (2023-02-14T10:12:10Z)
• Sparse high-dimensional linear regression with a partitioned empirical Bayes ECM algorithm [62.997667081978825]
  We propose a computationally efficient and powerful Bayesian approach for sparse high-dimensional linear regression. Minimal prior assumptions on the parameters are used through the use of plug-in empirical Bayes estimates. The proposed approach is implemented in the R package probe.
  arXiv  Detail & Related papers  (2022-09-16T19:15:50Z)
• Multi-objective hyperparameter optimization with performance uncertainty [62.997667081978825]
  This paper presents results on multi-objective hyperparameter optimization with uncertainty on the evaluation of Machine Learning algorithms. We combine the sampling strategy of Tree-structured Parzen Estimators (TPE) with the metamodel obtained after training a Gaussian Process Regression (GPR) with heterogeneous noise. Experimental results on three analytical test functions and three ML problems show the improvement over multi-objective TPE and GPR.
  arXiv  Detail & Related papers  (2022-09-09T14:58:43Z)
• Hierarchical Collaborative Hyper-parameter Tuning [0.0]
  Hyper- parameter tuning is among the most critical stages in building machine learning solutions. This paper demonstrates how multi-agent systems can be utilized to develop a distributed technique for determining near-optimal values.
  arXiv  Detail & Related papers  (2022-05-11T05:16:57Z)
• Online hyperparameter optimization by real-time recurrent learning [57.01871583756586]
  Our framework takes advantage of the analogy between hyperparameter optimization and parameter learning in neural networks (RNNs) It adapts a well-studied family of online learning algorithms for RNNs to tune hyperparameters and network parameters simultaneously. This procedure yields systematically better generalization performance compared to standard methods, at a fraction of wallclock time.
  arXiv  Detail & Related papers  (2021-02-15T19:36:18Z)
• A Study of Genetic Algorithms for Hyperparameter Optimization of Neural Networks in Machine Translation [0.0]
  We propose an automatic tuning method modeled after Darwin's Survival of the Fittest Theory via a Genetic Algorithm. Research results show that the proposed method, a GA, outperforms a random selection of hyper parameters.
  arXiv  Detail & Related papers  (2020-09-15T02:24:16Z)
• An Asymptotically Optimal Multi-Armed Bandit Algorithm and Hyperparameter Optimization [48.5614138038673]
  We propose an efficient and robust bandit-based algorithm called Sub-Sampling (SS) in the scenario of hyper parameter search evaluation. We also develop a novel hyper parameter optimization algorithm called BOSS. Empirical studies validate our theoretical arguments of SS and demonstrate the superior performance of BOSS on a number of applications.
  arXiv  Detail & Related papers  (2020-07-11T03:15:21Z)
• Automatically Learning Compact Quality-aware Surrogates for Optimization Problems [55.94450542785096]
  Solving optimization problems with unknown parameters requires learning a predictive model to predict the values of the unknown parameters and then solving the problem using these values. Recent work has shown that including the optimization problem as a layer in a complex training model pipeline results in predictions of iteration of unobserved decision making. We show that we can improve solution quality by learning a low-dimensional surrogate model of a large optimization problem.
  arXiv  Detail & Related papers  (2020-06-18T19:11:54Z)
• Better Trees: An empirical study on hyperparameter tuning of classification decision tree induction algorithms [5.4611430411491115]
  Decision Tree (DT) induction algorithms present high predictive performance and interpretable classification models. This paper investigates the effects of hyperparameter tuning for the two DT induction algorithms most often used, CART and C4.5. Experiments were carried out with different tuning strategies to induce models and to evaluate HPs' relevance using 94 classification datasets from OpenML.
  arXiv  Detail & Related papers  (2018-12-05T19:59:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.