Automatic Setting of DNN Hyper-Parameters by Mixing Bayesian Optimization and Tuning Rules

• URL: http://arxiv.org/abs/2006.02105v1
• Date: Wed, 3 Jun 2020 08:53:48 GMT
• Title: Automatic Setting of DNN Hyper-Parameters by Mixing Bayesian Optimization and Tuning Rules
• Authors: Michele Fraccaroli, Evelina Lamma, Fabrizio Riguzzi
• Abstract summary: We build a new algorithm for evaluating and analyzing the results of the network on the training and validation sets. We use a set of tuning rules to add new hyper-parameters and/or to reduce the hyper- parameter search space to select a better combination.
• Score: 0.6875312133832078
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep learning techniques play an increasingly important role in industrial and research environments due to their outstanding results. However, the large number of hyper-parameters to be set may lead to errors if they are set manually. The state-of-the-art hyper-parameters tuning methods are grid search, random search, and Bayesian Optimization. The first two methods are expensive because they try, respectively, all possible combinations and random combinations of hyper-parameters. Bayesian Optimization, instead, builds a surrogate model of the objective function, quantifies the uncertainty in the surrogate using Gaussian Process Regression and uses an acquisition function to decide where to sample the new set of hyper-parameters. This work faces the field of Hyper-Parameters Optimization (HPO). The aim is to improve Bayesian Optimization applied to Deep Neural Networks. For this goal, we build a new algorithm for evaluating and analyzing the results of the network on the training and validation sets and use a set of tuning rules to add new hyper-parameters and/or to reduce the hyper-parameter search space to select a better combination.

Related papers

• Parameter Optimization with Conscious Allocation (POCA) [4.478575931884855]
  Hyperband-based approaches to machine learning are among the most effective. We present. the new. Optimization with Conscious Allocation (POCA), a hyperband-based algorithm that adaptively allocates the inputted. budget to the hyperparameter configurations it generates. POCA finds strong configurations faster in both settings.
  arXiv  Detail & Related papers  (2023-12-29T00:13:55Z)
• Deep Ranking Ensembles for Hyperparameter Optimization [9.453554184019108]
  We present a novel method that meta-learns neural network surrogates optimized for ranking the configurations' performances while modeling their uncertainty via ensembling. In a large-scale experimental protocol comprising 12 baselines, 16 HPO search spaces and 86 datasets/tasks, we demonstrate that our method achieves new state-of-the-art results in HPO.
  arXiv  Detail & Related papers  (2023-03-27T13:52:40Z)
• Online Continuous Hyperparameter Optimization for Generalized Linear Contextual Bandits [55.03293214439741]
  In contextual bandits, an agent sequentially makes actions from a time-dependent action set based on past experience. We propose the first online continuous hyperparameter tuning framework for contextual bandits. We show that it could achieve a sublinear regret in theory and performs consistently better than all existing methods on both synthetic and real datasets.
  arXiv  Detail & Related papers  (2023-02-18T23:31:20Z)
• Towards Learning Universal Hyperparameter Optimizers with Transformers [57.35920571605559]
  We introduce the OptFormer, the first text-based Transformer HPO framework that provides a universal end-to-end interface for jointly learning policy and function prediction. Our experiments demonstrate that the OptFormer can imitate at least 7 different HPO algorithms, which can be further improved via its function uncertainty estimates.
  arXiv  Detail & Related papers  (2022-05-26T12:51:32Z)
• AUTOMATA: Gradient Based Data Subset Selection for Compute-Efficient Hyper-parameter Tuning [72.54359545547904]
  We propose a gradient-based subset selection framework for hyper- parameter tuning. We show that using gradient-based data subsets for hyper- parameter tuning achieves significantly faster turnaround times and speedups of 3$times$-30$times$.
  arXiv  Detail & Related papers  (2022-03-15T19:25:01Z)
• Towards Robust and Automatic Hyper-Parameter Tunning [39.04604349338802]
  We introduce a new class of HPO method and explore how the low-rank factorization of intermediate layers of a convolutional network can be used to define an analytical response surface. We quantify how this surface behaves as a surrogate to model performance and can be solved using a trust-region search algorithm, which we call autoHyper.
  arXiv  Detail & Related papers  (2021-11-28T05:27:34Z)
• Optimizing Large-Scale Hyperparameters via Automated Learning Algorithm [97.66038345864095]
  We propose a new hyperparameter optimization method with zeroth-order hyper-gradients (HOZOG) Specifically, we first formulate hyperparameter optimization as an A-based constrained optimization problem. Then, we use the average zeroth-order hyper-gradients to update hyper parameters.
  arXiv  Detail & Related papers  (2021-02-17T21:03:05Z)
• Online hyperparameter optimization by real-time recurrent learning [57.01871583756586]
  Our framework takes advantage of the analogy between hyperparameter optimization and parameter learning in neural networks (RNNs) It adapts a well-studied family of online learning algorithms for RNNs to tune hyperparameters and network parameters simultaneously. This procedure yields systematically better generalization performance compared to standard methods, at a fraction of wallclock time.
  arXiv  Detail & Related papers  (2021-02-15T19:36:18Z)
• An Asymptotically Optimal Multi-Armed Bandit Algorithm and Hyperparameter Optimization [48.5614138038673]
  We propose an efficient and robust bandit-based algorithm called Sub-Sampling (SS) in the scenario of hyper parameter search evaluation. We also develop a novel hyper parameter optimization algorithm called BOSS. Empirical studies validate our theoretical arguments of SS and demonstrate the superior performance of BOSS on a number of applications.
  arXiv  Detail & Related papers  (2020-07-11T03:15:21Z)
• Weighted Random Search for Hyperparameter Optimization [0.0]
  We introduce an improved version of Random Search (RS), used here for hyper parameter optimization of machine learning algorithms. We generate new values for each hyper parameter with a probability of change, unlike the standard RS. Within the same computational budget, our method yields better results than the standard RS.
  arXiv  Detail & Related papers  (2020-04-03T15:41:22Z)
• Weighted Random Search for CNN Hyperparameter Optimization [0.0]
  We introduce the weighted Random Search (WRS) method, a combination of Random Search (RS) and probabilistic greedy. The criterion is the classification accuracy achieved within the same number of tested combinations of hyperparameter values. According to our experiments, the WRS algorithm outperforms the other methods.
  arXiv  Detail & Related papers  (2020-03-30T09:40:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.