Rethinking of Encoder-based Warm-start Methods in Hyperparameter Optimization

• URL: http://arxiv.org/abs/2403.04720v4
• Date: Fri, 16 Aug 2024 12:08:03 GMT
• Title: Rethinking of Encoder-based Warm-start Methods in Hyperparameter Optimization
• Authors: Dawid Płudowski, Antoni Zajko, Anna Kozak, Katarzyna Woźnica,
• Abstract summary: We introduce a new approach for representation learning on tabular data based on Tomoharu Iwata and Atsutoshi Kumagai. We show that general representations may not suffice for some meta-tasks where requirements are not explicitly considered during extraction.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Effectively representing heterogeneous tabular datasets for meta-learning purposes remains an open problem. Previous approaches rely on predefined meta-features, for example, statistical measures or landmarkers. The emergence of dataset encoders opens new possibilities for the extraction of meta-features because they do not involve any handmade design. Moreover, they are proven to generate dataset representations with desired spatial properties. In this research, we evaluate an encoder-based approach to one of the most established meta-tasks - warm-starting of the Bayesian Hyperparameter Optimization. To broaden our analysis we introduce a new approach for representation learning on tabular data based on [Tomoharu Iwata and Atsutoshi Kumagai. Meta-learning from Tasks with Heterogeneous Attribute Spaces. In Advances in Neural Information Processing Systems, 2020]. The validation on over 100 datasets from UCI and an independent metaMIMIC set of datasets highlights the nuanced challenges in representation learning. We show that general representations may not suffice for some meta-tasks where requirements are not explicitly considered during extraction.

Related papers

• Minimally Supervised Learning using Topological Projections in Self-Organizing Maps [55.31182147885694]
  We introduce a semi-supervised learning approach based on topological projections in self-organizing maps (SOMs) Our proposed method first trains SOMs on unlabeled data and then a minimal number of available labeled data points are assigned to key best matching units (BMU) Our results indicate that the proposed minimally supervised model significantly outperforms traditional regression techniques.
  arXiv  Detail & Related papers  (2024-01-12T22:51:48Z)
• Self-Supervised Dataset Distillation for Transfer Learning [77.4714995131992]
  We propose a novel problem of distilling an unlabeled dataset into a set of small synthetic samples for efficient self-supervised learning (SSL) We first prove that a gradient of synthetic samples with respect to a SSL objective in naive bilevel optimization is textitbiased due to randomness originating from data augmentations or masking. We empirically validate the effectiveness of our method on various applications involving transfer learning.
  arXiv  Detail & Related papers  (2023-10-10T10:48:52Z)
• infoVerse: A Universal Framework for Dataset Characterization with Multidimensional Meta-information [68.76707843019886]
  infoVerse is a universal framework for dataset characterization. infoVerse captures multidimensional characteristics of datasets by incorporating various model-driven meta-information. In three real-world applications (data pruning, active learning, and data annotation), the samples chosen on infoVerse space consistently outperform strong baselines.
  arXiv  Detail & Related papers  (2023-05-30T18:12:48Z)
• Which is the best model for my data? [0.0]
  The proposed meta-learning approach relies on machine learning and involves four major steps. We present a collection of 62 meta-features that address the problem of information cancellation when aggregation measure values involving positive and negative measurements. We show that our meta-learning approach can correctly predict an optimal model for 91% of the synthetic datasets and for 87% of the real-world datasets.
  arXiv  Detail & Related papers  (2022-10-26T13:15:43Z)
• MARS: Meta-Learning as Score Matching in the Function Space [79.73213540203389]
  We present a novel approach to extracting inductive biases from a set of related datasets. We use functional Bayesian neural network inference, which views the prior as a process and performs inference in the function space. Our approach can seamlessly acquire and represent complex prior knowledge by metalearning the score function of the data-generating process.
  arXiv  Detail & Related papers  (2022-10-24T15:14:26Z)
• Towards Meta-learned Algorithm Selection using Implicit Fidelity Information [13.750624267664156]
  IMFAS produces informative landmarks, easily enriched by arbitrary meta-features at a low computational cost. We show it is able to beat Successive Halving with at most half the fidelity sequence during test time.
  arXiv  Detail & Related papers  (2022-06-07T09:14:24Z)
• T-METASET: Task-Aware Generation of Metamaterial Datasets by Diversity-Based Active Learning [14.668178146934588]
  We propose t-METASET: an intelligent data acquisition framework for task-aware dataset generation. We validate the proposed framework in three hypothetical deployment scenarios, which encompass general use, task-aware use, and tailorable use.
  arXiv  Detail & Related papers  (2022-02-21T22:46:49Z)
• Incremental Meta-Learning via Indirect Discriminant Alignment [118.61152684795178]
  We develop a notion of incremental learning during the meta-training phase of meta-learning. Our approach performs favorably at test time as compared to training a model with the full meta-training set.
  arXiv  Detail & Related papers  (2020-02-11T01:39:12Z)
• Meta-learning framework with applications to zero-shot time-series forecasting [82.61728230984099]
  This work provides positive evidence using a broad meta-learning framework. residual connections act as a meta-learning adaptation mechanism. We show that it is viable to train a neural network on a source TS dataset and deploy it on a different target TS dataset without retraining.
  arXiv  Detail & Related papers  (2020-02-07T16:39:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.