Impact of Training Instance Selection on Automated Algorithm Selection Models for Numerical Black-box Optimization

• URL: http://arxiv.org/abs/2404.07539v1
• Date: Thu, 11 Apr 2024 08:03:53 GMT
• Title: Impact of Training Instance Selection on Automated Algorithm Selection Models for Numerical Black-box Optimization
• Authors: Konstantin Dietrich, Diederick Vermetten, Carola Doerr, Pascal Kerschke,
• Abstract summary: We show that MA-BBOB-generated functions can be an ideal testbed for automated machine learning methods. We analyze the potential gains from AAS by studying performance complementarity within a set of eight algorithms. We show that simply using the BBOB component functions for training yields poor test performance.
• Score: 0.40498500266986387
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The recently proposed MA-BBOB function generator provides a way to create numerical black-box benchmark problems based on the well-established BBOB suite. Initial studies on this generator highlighted its ability to smoothly transition between the component functions, both from a low-level landscape feature perspective, as well as with regard to algorithm performance. This suggests that MA-BBOB-generated functions can be an ideal testbed for automated machine learning methods, such as automated algorithm selection (AAS). In this paper, we generate 11800 functions in dimensions $d=2$ and $d=5$, respectively, and analyze the potential gains from AAS by studying performance complementarity within a set of eight algorithms. We combine this performance data with exploratory landscape features to create an AAS pipeline that we use to investigate how to efficiently select training sets within this space. We show that simply using the BBOB component functions for training yields poor test performance, while the ranking between uniformly chosen and diversity-based training sets strongly depends on the distribution of the test set.

Related papers

• Landscape-Aware Automated Algorithm Configuration using Multi-output Mixed Regression and Classification [0.01649298969786889]
  We investigate the potential of randomly generated functions (RGF) for the model training. We focus on automated algorithm configuration (AAC) We analyze the performance of dense neural network (NN) models in handling the mixed regression and classification tasks.
  arXiv  Detail & Related papers  (2024-09-02T20:04:41Z)
• Reinforced In-Context Black-Box Optimization [64.25546325063272]
  RIBBO is a method to reinforce-learn a BBO algorithm from offline data in an end-to-end fashion. RIBBO employs expressive sequence models to learn the optimization histories produced by multiple behavior algorithms and tasks. Central to our method is to augment the optimization histories with textitregret-to-go tokens, which are designed to represent the performance of an algorithm based on cumulative regret over the future part of the histories.
  arXiv  Detail & Related papers  (2024-02-27T11:32:14Z)
• MA-BBOB: A Problem Generator for Black-Box Optimization Using Affine Combinations and Shifts [1.2617078020344619]
  We present the MA-BBOB function generator, which uses the BBOB suite as component functions in an affine combination. We show a potential use-case of MA-BBOB in generating a wide set of training and testing data for algorithm selectors.
  arXiv  Detail & Related papers  (2023-12-18T10:23:09Z)
• MA-BBOB: Many-Affine Combinations of BBOB Functions for Evaluating AutoML Approaches in Noiseless Numerical Black-Box Optimization Contexts [0.8258451067861933]
  (MA-)BBOB is built on the publicly available IOHprofiler platform. It provides access to the interactive IOHanalyzer module for performance analysis and visualization, and enables comparisons with the rich and growing data collection available for the (MA-)BBOB functions.
  arXiv  Detail & Related papers  (2023-06-18T19:32:12Z)
• DynamoRep: Trajectory-Based Population Dynamics for Classification of Black-box Optimization Problems [0.755972004983746]
  We propose a feature extraction method that describes the trajectories of optimization algorithms using simple statistics. We demonstrate that the proposed DynamoRep features capture enough information to identify the problem class on which the optimization algorithm is running.
  arXiv  Detail & Related papers  (2023-06-08T06:57:07Z)
• Active Finetuning: Exploiting Annotation Budget in the Pretraining-Finetuning Paradigm [132.9949120482274]
  This paper focuses on the selection of samples for annotation in the pretraining-finetuning paradigm. We propose a novel method called ActiveFT for active finetuning task to select a subset of data distributing similarly with the entire unlabeled pool. Extensive experiments show the leading performance and high efficiency of ActiveFT superior to baselines on both image classification and semantic segmentation.
  arXiv  Detail & Related papers  (2023-03-25T07:17:03Z)
• Improved Algorithms for Neural Active Learning [74.89097665112621]
  We improve the theoretical and empirical performance of neural-network(NN)-based active learning algorithms for the non-parametric streaming setting. We introduce two regret metrics by minimizing the population loss that are more suitable in active learning than the one used in state-of-the-art (SOTA) related work.
  arXiv  Detail & Related papers  (2022-10-02T05:03:38Z)
• Surrogate modeling for Bayesian optimization beyond a single Gaussian process [62.294228304646516]
  We propose a novel Bayesian surrogate model to balance exploration with exploitation of the search space. To endow function sampling with scalability, random feature-based kernel approximation is leveraged per GP model. To further establish convergence of the proposed EGP-TS to the global optimum, analysis is conducted based on the notion of Bayesian regret.
  arXiv  Detail & Related papers  (2022-05-27T16:43:10Z)
• Compactness Score: A Fast Filter Method for Unsupervised Feature Selection [66.84571085643928]
  We propose a fast unsupervised feature selection method, named as, Compactness Score (CSUFS) to select desired features. Our proposed algorithm seems to be more accurate and efficient compared with existing algorithms.
  arXiv  Detail & Related papers  (2022-01-31T13:01:37Z)
• Stepwise Model Selection for Sequence Prediction via Deep Kernel Learning [100.83444258562263]
  We propose a novel Bayesian optimization (BO) algorithm to tackle the challenge of model selection in this setting. In order to solve the resulting multiple black-box function optimization problem jointly and efficiently, we exploit potential correlations among black-box functions. We are the first to formulate the problem of stepwise model selection (SMS) for sequence prediction, and to design and demonstrate an efficient joint-learning algorithm for this purpose.
  arXiv  Detail & Related papers  (2020-01-12T09:42:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.