MOFA: Modular Factorial Design for Hyperparameter Optimization

• URL: http://arxiv.org/abs/2011.09545v2
• Date: Thu, 3 Jun 2021 09:07:44 GMT
• Title: MOFA: Modular Factorial Design for Hyperparameter Optimization
• Authors: Bo Xiong, Yimin Huang, Hanrong Ye, Steffen Staab, Zhenguo Li
• Abstract summary: MOdular FActorial Design (MOFA) is a novel HPO method that exploits evaluation results through factorial analysis. We prove that the inference of MOFA achieves higher confidence than other sampling schemes. Empirical results show that MOFA achieves better effectiveness and efficiency compared with state-of-the-art methods.
• Score: 47.779983311833014
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper presents a novel and lightweight hyperparameter optimization (HPO) method, MOdular FActorial Design (MOFA). MOFA pursues several rounds of HPO, where each round alternates between exploration of hyperparameter space by factorial design and exploitation of evaluation results by factorial analysis. Each round first explores the configuration space by constructing a low-discrepancy set of hyperparameters that cover this space well while de-correlating hyperparameters, and then exploits evaluation results through factorial analysis that determines which hyperparameters should be further explored and which should become fixed in the next round. We prove that the inference of MOFA achieves higher confidence than other sampling schemes. Each individual round is highly parallelizable and hence offers major improvements of efficiency compared to model-based methods. Empirical results show that MOFA achieves better effectiveness and efficiency compared with state-of-the-art methods.

Related papers

• Efficient and Robust Parameter Optimization of the Unitary Coupled-Cluster Ansatz [4.607081302947026]
  We propose sequential optimization with approximate parabola (SOAP) for parameter optimization of unitary coupled-cluster ansatz on quantum computers. Numerical benchmark studies on molecular systems demonstrate that SOAP achieves significantly faster convergence and greater robustness to noise. SOAP is further validated through experiments on a superconducting quantum computer using a 2-qubit model system.
  arXiv  Detail & Related papers  (2024-01-10T03:30:39Z)
• A Dynamic-Neighbor Particle Swarm Optimizer for Accurate Latent Factor Analysis [8.451827165005993]
  The performance of an LFA model heavily rely on its optimization process. Some prior studies employ the Particle Swarm Optimization to enhance an LFA model's optimization process. This paper proposes a Dynamic-neighbor-cooperated Hierarchical PSO-enhanced LFA model with two-fold main ideas.
  arXiv  Detail & Related papers  (2023-02-23T12:03:59Z)
• On the Effectiveness of Parameter-Efficient Fine-Tuning [79.6302606855302]
  Currently, many research works propose to only fine-tune a small portion of the parameters while keeping most of the parameters shared across different tasks. We show that all of the methods are actually sparse fine-tuned models and conduct a novel theoretical analysis of them. Despite the effectiveness of sparsity grounded by our theory, it still remains an open problem of how to choose the tunable parameters.
  arXiv  Detail & Related papers  (2022-11-28T17:41:48Z)
• A General Framework for Sample-Efficient Function Approximation in Reinforcement Learning [132.45959478064736]
  We propose a general framework that unifies model-based and model-free reinforcement learning. We propose a novel estimation function with decomposable structural properties for optimization-based exploration. Under our framework, a new sample-efficient algorithm namely OPtimization-based ExploRation with Approximation (OPERA) is proposed.
  arXiv  Detail & Related papers  (2022-09-30T17:59:16Z)
• Sparse high-dimensional linear regression with a partitioned empirical Bayes ECM algorithm [62.997667081978825]
  We propose a computationally efficient and powerful Bayesian approach for sparse high-dimensional linear regression. Minimal prior assumptions on the parameters are used through the use of plug-in empirical Bayes estimates. The proposed approach is implemented in the R package probe.
  arXiv  Detail & Related papers  (2022-09-16T19:15:50Z)
• Multi-objective hyperparameter optimization with performance uncertainty [62.997667081978825]
  This paper presents results on multi-objective hyperparameter optimization with uncertainty on the evaluation of Machine Learning algorithms. We combine the sampling strategy of Tree-structured Parzen Estimators (TPE) with the metamodel obtained after training a Gaussian Process Regression (GPR) with heterogeneous noise. Experimental results on three analytical test functions and three ML problems show the improvement over multi-objective TPE and GPR.
  arXiv  Detail & Related papers  (2022-09-09T14:58:43Z)
• Optimizing Training Trajectories in Variational Autoencoders via Latent Bayesian Optimization Approach [0.0]
  Unsupervised and semi-supervised ML methods have become widely adopted across multiple areas of physics, chemistry, and materials sciences. We propose a latent Bayesian optimization (zBO) approach for the hyper parameter trajectory optimization for the unsupervised and semi-supervised ML. We demonstrate an application of this method for finding joint discrete and continuous rotationally invariant representations for MNIST and experimental data of a plasmonic nanoparticles material system.
  arXiv  Detail & Related papers  (2022-06-30T23:41:47Z)
• Towards Learning Universal Hyperparameter Optimizers with Transformers [57.35920571605559]
  We introduce the OptFormer, the first text-based Transformer HPO framework that provides a universal end-to-end interface for jointly learning policy and function prediction. Our experiments demonstrate that the OptFormer can imitate at least 7 different HPO algorithms, which can be further improved via its function uncertainty estimates.
  arXiv  Detail & Related papers  (2022-05-26T12:51:32Z)
• Hierarchical Collaborative Hyper-parameter Tuning [0.0]
  Hyper- parameter tuning is among the most critical stages in building machine learning solutions. This paper demonstrates how multi-agent systems can be utilized to develop a distributed technique for determining near-optimal values.
  arXiv  Detail & Related papers  (2022-05-11T05:16:57Z)
• Supervising the Multi-Fidelity Race of Hyperparameter Configurations [22.408069485293666]
  We introduce DyHPO, a Bayesian Optimization method that learns to decide which hyperparameter configuration to train further in a race among all feasible configurations. We demonstrate the significant superiority of DyHPO against state-of-the-art hyperparameter optimization methods through large-scale experiments.
  arXiv  Detail & Related papers  (2022-02-20T10:28:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.