Hybrid Parameter Search and Dynamic Model Selection for Mixed-Variable Bayesian Optimization

• URL: http://arxiv.org/abs/2206.01409v4
• Date: Fri, 19 Jan 2024 00:23:28 GMT
• Title: Hybrid Parameter Search and Dynamic Model Selection for Mixed-Variable Bayesian Optimization
• Authors: Hengrui Luo, Younghyun Cho, James W. Demmel, Xiaoye S. Li, Yang Liu
• Abstract summary: We propose a new type of hybrid model for Bayesian optimization (BO) adept at managing mixed variables. Our proposed new hybrid models (named hybridM) merge the Monte Carlo Tree Search structure (MCTS) for categorical variables with Gaussian Processes (GP) for continuous ones. Our innovations, including dynamic online kernel selection in the surrogate modeling phase, position our hybrid models as an advancement in mixed-variable surrogate models.
• Score: 6.204805504959941
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper presents a new type of hybrid model for Bayesian optimization (BO) adept at managing mixed variables, encompassing both quantitative (continuous and integer) and qualitative (categorical) types. Our proposed new hybrid models (named hybridM) merge the Monte Carlo Tree Search structure (MCTS) for categorical variables with Gaussian Processes (GP) for continuous ones. hybridM leverages the upper confidence bound tree search (UCTS) for MCTS strategy, showcasing the tree architecture's integration into Bayesian optimization. Our innovations, including dynamic online kernel selection in the surrogate modeling phase and a unique UCTS search strategy, position our hybrid models as an advancement in mixed-variable surrogate models. Numerical experiments underscore the superiority of hybrid models, highlighting their potential in Bayesian optimization.

Related papers

• Deep Generative Clustering with VAEs and Expectation-Maximization [1.8416014644193066]
  We propose a novel deep clustering method that integrates Variational Autoencoders (VAEs) into the Expectation-Maximization framework. Our approach models the probability distribution of each cluster with a VAE and alternates between updating model parameters. This enables effective clustering and generation of new samples from each cluster.
  arXiv  Detail & Related papers  (2025-01-13T14:26:39Z)
• Be More Diverse than the Most Diverse: Online Selection of Diverse Mixtures of Generative Models [33.04472814852163]
  In this work, we explore the selection of a mixture of multiple generative models. We propose an online learning approach called Mixture Upper Confidence Bound (Mixture-UCB)
  arXiv  Detail & Related papers  (2024-12-23T14:48:17Z)
• STAR: Synthesis of Tailored Architectures [61.080157488857516]
  We propose a new approach for the synthesis of tailored architectures (STAR) Our approach combines a novel search space based on the theory of linear input-varying systems, supporting a hierarchical numerical encoding into architecture genomes. STAR genomes are automatically refined and recombined with gradient-free, evolutionary algorithms to optimize for multiple model quality and efficiency metrics. Using STAR, we optimize large populations of new architectures, leveraging diverse computational units and interconnection patterns, improving over highly-optimized Transformers and striped hybrid models on the frontier of quality, parameter size, and inference cache for autoregressive language modeling.
  arXiv  Detail & Related papers  (2024-11-26T18:42:42Z)
• Automatically Learning Hybrid Digital Twins of Dynamical Systems [56.69628749813084]
  Digital Twins (DTs) simulate the states and temporal dynamics of real-world systems. DTs often struggle to generalize to unseen conditions in data-scarce settings. In this paper, we propose an evolutionary algorithm ($textbfHDTwinGen$) to autonomously propose, evaluate, and optimize HDTwins.
  arXiv  Detail & Related papers  (2024-10-31T07:28:22Z)
• Model-GLUE: Democratized LLM Scaling for A Large Model Zoo in the Wild [84.57103623507082]
  This paper introduces Model-GLUE, a holistic Large Language Models scaling guideline. We benchmark existing scaling techniques, especially selective merging, and variants of mixture. We then formulate an optimal strategy for the selection and aggregation of a heterogeneous model zoo. Our methodology involves the clustering of mergeable models and optimal merging strategy selection, and the integration of clusters.
  arXiv  Detail & Related papers  (2024-10-07T15:55:55Z)
• Hybrid State Space-based Learning for Sequential Data Prediction with Joint Optimization [0.0]
  We introduce a hybrid model that mitigates, via a joint mechanism, the need for domain-specific feature engineering issues of conventional nonlinear prediction models. We achieve this by introducing novel state space representations for the base models, which are then combined to provide a full state space representation of the hybrid or the ensemble. Due to such novel combination and joint optimization, we demonstrate significant improvements in widely publicized real life competition datasets.
  arXiv  Detail & Related papers  (2023-09-19T12:00:28Z)
• Hybrid additive modeling with partial dependence for supervised regression and dynamical systems forecasting [5.611231523622238]
  We introduce a new hybrid training approach based on partial dependence, which removes the need for intricate regularization. We compare, on both synthetic and real regression problems, several approaches for training such hybrid models. Experiments are carried out with different types of machine learning models, including tree-based models and artificial neural networks.
  arXiv  Detail & Related papers  (2023-07-05T12:13:56Z)
• AI-Empowered Hybrid MIMO Beamforming [85.48860461696417]
  Hybrid multiple-input multiple-output (MIMO) systems implement part of their beamforming in analog and part in digital. Recent years have witnessed a growing interest in using data-aided artificial intelligence (AI) tools for hybrid beamforming design. This article reviews candidate strategies to leverage data to improve real-time hybrid beamforming design.
  arXiv  Detail & Related papers  (2023-03-03T06:04:20Z)
• Applying Autonomous Hybrid Agent-based Computing to Difficult Optimization Problems [56.821213236215634]
  This paper focuses on a proposed hybrid version of the EMAS. It covers selection and introduction of a number of hybrid operators and defining rules for starting the hybrid steps of the main algorithm. Those hybrid steps leverage existing, well-known and proven to be efficient metaheuristics, and integrate their results into the main algorithm.
  arXiv  Detail & Related papers  (2022-10-24T13:28:35Z)
• A Pareto-optimal compositional energy-based model for sampling and optimization of protein sequences [55.25331349436895]
  Deep generative models have emerged as a popular machine learning-based approach for inverse problems in the life sciences. These problems often require sampling new designs that satisfy multiple properties of interest in addition to learning the data distribution.
  arXiv  Detail & Related papers  (2022-10-19T19:04:45Z)
• A hybrid ensemble method with negative correlation learning for regression [2.8484009470171943]
  This study automatically selects and weights sub-models from a heterogeneous model pool. It solves an optimization problem using an interior-point filtering linear-search algorithm. The value of this study lies in its ease of use and effectiveness, allowing the hybrid ensemble to embrace diversity and accuracy.
  arXiv  Detail & Related papers  (2021-04-06T06:45:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.