Speeding Up Multi-Objective Hyperparameter Optimization by Task Similarity-Based Meta-Learning for the Tree-Structured Parzen Estimator

• URL: http://arxiv.org/abs/2212.06751v5
• Date: Wed, 31 May 2023 06:40:52 GMT
• Title: Speeding Up Multi-Objective Hyperparameter Optimization by Task Similarity-Based Meta-Learning for the Tree-Structured Parzen Estimator
• Authors: Shuhei Watanabe, Noor Awad, Masaki Onishi, Frank Hutter
• Abstract summary: In this paper, we extend TPE's acquisition function to the meta-learning setting using a task similarity defined by the overlap of top domains between tasks. In the experiments, we demonstrate that our method speeds up MO-TPE on tabular HPO benchmarks and attains state-of-the-art performance.
• Score: 37.553558410770314
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Hyperparameter optimization (HPO) is a vital step in improving performance in deep learning (DL). Practitioners are often faced with the trade-off between multiple criteria, such as accuracy and latency. Given the high computational needs of DL and the growing demand for efficient HPO, the acceleration of multi-objective (MO) optimization becomes ever more important. Despite the significant body of work on meta-learning for HPO, existing methods are inapplicable to MO tree-structured Parzen estimator (MO-TPE), a simple yet powerful MO-HPO algorithm. In this paper, we extend TPE's acquisition function to the meta-learning setting using a task similarity defined by the overlap of top domains between tasks. We also theoretically analyze and address the limitations of our task similarity. In the experiments, we demonstrate that our method speeds up MO-TPE on tabular HPO benchmarks and attains state-of-the-art performance. Our method was also validated externally by winning the AutoML 2022 competition on "Multiobjective Hyperparameter Optimization for Transformers".

Related papers

• EPS-MoE: Expert Pipeline Scheduler for Cost-Efficient MoE Inference [49.94169109038806]
  This paper introduces EPS-MoE, a novel expert pipeline scheduler for MoE. Our results demonstrate an average 21% improvement in prefill throughput over existing parallel inference methods.
  arXiv  Detail & Related papers  (2024-10-16T05:17:49Z)
• Large Language Model Agent for Hyper-Parameter Optimization [30.560250427498243]
  We introduce a novel paradigm leveraging Large Language Models (LLMs) to automate hyperparameter optimization across diverse machine learning tasks. AgentHPO processes the task information autonomously, conducts experiments with specific hyper parameters, and iteratively optimize them. This human-like optimization process largely reduces the number of required trials, simplifies the setup process, and enhances interpretability and user trust.
  arXiv  Detail & Related papers  (2024-02-02T20:12:05Z)
• Interactive Hyperparameter Optimization in Multi-Objective Problems via Preference Learning [65.51668094117802]
  We propose a human-centered interactive HPO approach tailored towards multi-objective machine learning (ML) Instead of relying on the user guessing the most suitable indicator for their needs, our approach automatically learns an appropriate indicator.
  arXiv  Detail & Related papers  (2023-09-07T09:22:05Z)
• MO-DEHB: Evolutionary-based Hyperband for Multi-Objective Optimization [30.54386890506418]
  MO-DEHB is an effective and flexible multi-objective (MO) that extends the recent evolutionary Hyperband method DEHB. A comparative study against state-of-the-art MOs demonstrates that MO-DEHB clearly achieves the best performance across our 15 benchmarks.
  arXiv  Detail & Related papers  (2023-05-08T06:53:40Z)
• Two-step hyperparameter optimization method: Accelerating hyperparameter search by using a fraction of a training dataset [0.15420205433587747]
  We present a two-step HPO method as a strategic solution to curbing computational demands and wait times. We present our recent application of the two-step HPO method to the development of neural network emulators for aerosol activation.
  arXiv  Detail & Related papers  (2023-02-08T02:38:26Z)
• c-TPE: Tree-structured Parzen Estimator with Inequality Constraints for Expensive Hyperparameter Optimization [45.67326752241075]
  We propose constrained TPE (c-TPE), an extension of the widely-used versatile Bayesian optimization method, tree-structured Parzen estimator (TPE) to handle these constraints. Our proposed extension goes beyond a simple combination of an existing acquisition function and the original TPE, and instead includes modifications that address issues that cause poor performance. In the experiments, we demonstrate that c-TPE exhibits the best average rank performance among existing methods with statistical significance on 81 expensive HPO with inequality constraints.
  arXiv  Detail & Related papers  (2022-11-26T00:25:11Z)
• Multi-objective hyperparameter optimization with performance uncertainty [62.997667081978825]
  This paper presents results on multi-objective hyperparameter optimization with uncertainty on the evaluation of Machine Learning algorithms. We combine the sampling strategy of Tree-structured Parzen Estimators (TPE) with the metamodel obtained after training a Gaussian Process Regression (GPR) with heterogeneous noise. Experimental results on three analytical test functions and three ML problems show the improvement over multi-objective TPE and GPR.
  arXiv  Detail & Related papers  (2022-09-09T14:58:43Z)
• Towards Learning Universal Hyperparameter Optimizers with Transformers [57.35920571605559]
  We introduce the OptFormer, the first text-based Transformer HPO framework that provides a universal end-to-end interface for jointly learning policy and function prediction. Our experiments demonstrate that the OptFormer can imitate at least 7 different HPO algorithms, which can be further improved via its function uncertainty estimates.
  arXiv  Detail & Related papers  (2022-05-26T12:51:32Z)
• Self-Evolutionary Optimization for Pareto Front Learning [34.17125297176668]
  Multi-objective optimization (MOO) approaches have been proposed for multitasking problems. Recent MOO methods approximate multiple optimal solutions (Pareto front) with a single unified model. We show that PFL can be re-formulated into another MOO problem with multiple objectives, each of which corresponds to different preference weights for the tasks.
  arXiv  Detail & Related papers  (2021-10-07T13:38:57Z)
• DHA: End-to-End Joint Optimization of Data Augmentation Policy, Hyper-parameter and Architecture [81.82173855071312]
  We propose an end-to-end solution that integrates the AutoML components and returns a ready-to-use model at the end of the search. Dha achieves state-of-the-art (SOTA) results on various datasets, especially 77.4% accuracy on ImageNet with cell based search space.
  arXiv  Detail & Related papers  (2021-09-13T08:12:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.