Optimizing Large-Scale Hyperparameters via Automated Learning Algorithm
- URL: http://arxiv.org/abs/2102.09026v1
- Date: Wed, 17 Feb 2021 21:03:05 GMT
- Title: Optimizing Large-Scale Hyperparameters via Automated Learning Algorithm
- Authors: Bin Gu, Guodong Liu, Yanfu Zhang, Xiang Geng, Heng Huang
- Abstract summary: We propose a new hyperparameter optimization method with zeroth-order hyper-gradients (HOZOG)
Specifically, we first formulate hyperparameter optimization as an A-based constrained optimization problem.
Then, we use the average zeroth-order hyper-gradients to update hyper parameters.
- Score: 97.66038345864095
- License: http://creativecommons.org/publicdomain/zero/1.0/
- Abstract: Modern machine learning algorithms usually involve tuning multiple (from one
to thousands) hyperparameters which play a pivotal role in terms of model
generalizability. Black-box optimization and gradient-based algorithms are two
dominant approaches to hyperparameter optimization while they have totally
distinct advantages. How to design a new hyperparameter optimization technique
inheriting all benefits from both approaches is still an open problem. To
address this challenging problem, in this paper, we propose a new
hyperparameter optimization method with zeroth-order hyper-gradients (HOZOG).
Specifically, we first exactly formulate hyperparameter optimization as an
A-based constrained optimization problem, where A is a black-box optimization
algorithm (such as deep neural network). Then, we use the average zeroth-order
hyper-gradients to update hyperparameters. We provide the feasibility analysis
of using HOZOG to achieve hyperparameter optimization. Finally, the
experimental results on three representative hyperparameter (the size is from 1
to 1250) optimization tasks demonstrate the benefits of HOZOG in terms of
simplicity, scalability, flexibility, effectiveness and efficiency compared
with the state-of-the-art hyperparameter optimization methods.
Related papers
- End-to-End Learning for Fair Multiobjective Optimization Under
Uncertainty [55.04219793298687]
The Predict-Then-Forecast (PtO) paradigm in machine learning aims to maximize downstream decision quality.
This paper extends the PtO methodology to optimization problems with nondifferentiable Ordered Weighted Averaging (OWA) objectives.
It shows how optimization of OWA functions can be effectively integrated with parametric prediction for fair and robust optimization under uncertainty.
arXiv Detail & Related papers (2024-02-12T16:33:35Z) - A Globally Convergent Gradient-based Bilevel Hyperparameter Optimization
Method [0.0]
We propose a gradient-based bilevel method for solving the hyperparameter optimization problem.
We show that the proposed method converges with lower computation and leads to models that generalize better on the testing set.
arXiv Detail & Related papers (2022-08-25T14:25:16Z) - A Comparative study of Hyper-Parameter Optimization Tools [2.6097538974670935]
We compare the performance of four python libraries, namely Optuna, Hyperopt, Optunity, and sequential model algorithm configuration (SMAC)
We found that Optuna has better performance for CASH problem and NeurIPS black-box optimization challenge.
arXiv Detail & Related papers (2022-01-17T14:49:36Z) - HyP-ABC: A Novel Automated Hyper-Parameter Tuning Algorithm Using
Evolutionary Optimization [1.6114012813668934]
We propose HyP-ABC, an automatic hybrid hyper-parameter optimization algorithm using the modified artificial bee colony approach.
Compared to the state-of-the-art techniques, HyP-ABC is more efficient and has a limited number of parameters to be tuned.
arXiv Detail & Related papers (2021-09-11T16:45:39Z) - Implicit differentiation for fast hyperparameter selection in non-smooth
convex learning [87.60600646105696]
We study first-order methods when the inner optimization problem is convex but non-smooth.
We show that the forward-mode differentiation of proximal gradient descent and proximal coordinate descent yield sequences of Jacobians converging toward the exact Jacobian.
arXiv Detail & Related papers (2021-05-04T17:31:28Z) - Online hyperparameter optimization by real-time recurrent learning [57.01871583756586]
Our framework takes advantage of the analogy between hyperparameter optimization and parameter learning in neural networks (RNNs)
It adapts a well-studied family of online learning algorithms for RNNs to tune hyperparameters and network parameters simultaneously.
This procedure yields systematically better generalization performance compared to standard methods, at a fraction of wallclock time.
arXiv Detail & Related papers (2021-02-15T19:36:18Z) - Adaptive pruning-based optimization of parameterized quantum circuits [62.997667081978825]
Variisy hybrid quantum-classical algorithms are powerful tools to maximize the use of Noisy Intermediate Scale Quantum devices.
We propose a strategy for such ansatze used in variational quantum algorithms, which we call "Efficient Circuit Training" (PECT)
Instead of optimizing all of the ansatz parameters at once, PECT launches a sequence of variational algorithms.
arXiv Detail & Related papers (2020-10-01T18:14:11Z) - A Gradient-based Bilevel Optimization Approach for Tuning
Hyperparameters in Machine Learning [0.0]
We propose a bilevel solution method for solving the hyperparameter optimization problem.
The proposed method is general and can be easily applied to any class of machine learning algorithms.
We discuss the theory behind the proposed algorithm and perform extensive computational study on two datasets.
arXiv Detail & Related papers (2020-07-21T18:15:08Z) - A Modified Bayesian Optimization based Hyper-Parameter Tuning Approach
for Extreme Gradient Boosting [3.655021726150369]
One of the ways to perform Hyper- optimization is by manual search but that is time consuming.
Some of the common approaches for performing Hyper- optimization are Grid search Random search and Bayesian optimization using Hyperopt.
We find that the Randomized-Hyperopt performs better than the other three conventional methods for hyper-paramter optimization of XGBoost.
arXiv Detail & Related papers (2020-04-10T14:09:54Z) - Cross Entropy Hyperparameter Optimization for Constrained Problem
Hamiltonians Applied to QAOA [68.11912614360878]
Hybrid quantum-classical algorithms such as Quantum Approximate Optimization Algorithm (QAOA) are considered as one of the most encouraging approaches for taking advantage of near-term quantum computers in practical applications.
Such algorithms are usually implemented in a variational form, combining a classical optimization method with a quantum machine to find good solutions to an optimization problem.
In this study we apply a Cross-Entropy method to shape this landscape, which allows the classical parameter to find better parameters more easily and hence results in an improved performance.
arXiv Detail & Related papers (2020-03-11T13:52:41Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.