MO-DEHB: Evolutionary-based Hyperband for Multi-Objective Optimization
- URL: http://arxiv.org/abs/2305.04502v2
- Date: Thu, 11 May 2023 07:32:09 GMT
- Title: MO-DEHB: Evolutionary-based Hyperband for Multi-Objective Optimization
- Authors: Noor Awad, Ayushi Sharma, Philipp Muller, Janek Thomas and Frank
Hutter
- Abstract summary: MO-DEHB is an effective and flexible multi-objective (MO) that extends the recent evolutionary Hyperband method DEHB.
A comparative study against state-of-the-art MOs demonstrates that MO-DEHB clearly achieves the best performance across our 15 benchmarks.
- Score: 30.54386890506418
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Hyperparameter optimization (HPO) is a powerful technique for automating the
tuning of machine learning (ML) models. However, in many real-world
applications, accuracy is only one of multiple performance criteria that must
be considered. Optimizing these objectives simultaneously on a complex and
diverse search space remains a challenging task. In this paper, we propose
MO-DEHB, an effective and flexible multi-objective (MO) optimizer that extends
the recent evolutionary Hyperband method DEHB. We validate the performance of
MO-DEHB using a comprehensive suite of 15 benchmarks consisting of diverse and
challenging MO problems, including HPO, neural architecture search (NAS), and
joint NAS and HPO, with objectives including accuracy, latency and algorithmic
fairness. A comparative study against state-of-the-art MO optimizers
demonstrates that MO-DEHB clearly achieves the best performance across our 15
benchmarks.
Related papers
- Decoding-Time Language Model Alignment with Multiple Objectives [88.64776769490732]
Existing methods primarily focus on optimizing LMs for a single reward function, limiting their adaptability to varied objectives.
Here, we propose $textbfmulti-objective decoding (MOD)$, a decoding-time algorithm that outputs the next token from a linear combination of predictions.
We show why existing approaches can be sub-optimal even in natural settings and obtain optimality guarantees for our method.
arXiv Detail & Related papers (2024-06-27T02:46:30Z) - Intuition-aware Mixture-of-Rank-1-Experts for Parameter Efficient Finetuning [50.73666458313015]
Large Language Models (LLMs) have demonstrated significant potential in performing multiple tasks in multimedia applications.
MoE has been emerged as a promising solution with its sparse architecture for effective task decoupling.
Intuition-MoR1E achieves superior efficiency and 2.15% overall accuracy improvement across 14 public datasets.
arXiv Detail & Related papers (2024-04-13T12:14:58Z) - Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct
Preference Optimization [78.50294936259026]
We present Multi-Objective Direct Preference Optimization (MODPO) for multiple alignment objectives with minimal overheads.
MODPO folds language modeling directly into reward modeling, training LMs as implicit collective reward models (cRMs) that combine all objectives with specific weightings.
While theoretically guaranteed to produce the same optimal solutions as MORLHF, MODPO is practically more stable and computationally efficient.
arXiv Detail & Related papers (2023-10-05T17:35:26Z) - Multi-Objective Population Based Training [62.997667081978825]
Population Based Training (PBT) is an efficient hyperparameter optimization algorithm.
In this work, we introduce a multi-objective version of PBT, MO-PBT.
arXiv Detail & Related papers (2023-06-02T10:54:24Z) - BOtied: Multi-objective Bayesian optimization with tied multivariate ranks [33.414682601242006]
In this paper, we show a natural connection between non-dominated solutions and the extreme quantile of the joint cumulative distribution function.
Motivated by this link, we propose the Pareto-compliant CDF indicator and the associated acquisition function, BOtied.
Our experiments on a variety of synthetic and real-world problems demonstrate that BOtied outperforms state-of-the-art MOBO acquisition functions.
arXiv Detail & Related papers (2023-06-01T04:50:06Z) - Multi objective Fitness Dependent Optimizer Algorithm [19.535715565093764]
This paper proposes the multi objective variant of the recently introduced fitness dependent (FDO)
The algorithm is called a Multi objective Fitness Dependent (MOFDO) and is equipped with all five types of knowledge (situational, normative, topographical, domain, and historical knowledge) as in FDO.
It is observed that the proposed algorithm successfully provides a wide variety of well-distributed feasible solutions, which enable the decision-makers to have more applicable-comfort choices to consider.
arXiv Detail & Related papers (2023-01-26T06:33:53Z) - Speeding Up Multi-Objective Hyperparameter Optimization by Task
Similarity-Based Meta-Learning for the Tree-Structured Parzen Estimator [37.553558410770314]
In this paper, we extend TPE's acquisition function to the meta-learning setting using a task similarity defined by the overlap of top domains between tasks.
In the experiments, we demonstrate that our method speeds up MO-TPE on tabular HPO benchmarks and attains state-of-the-art performance.
arXiv Detail & Related papers (2022-12-13T17:33:02Z) - Multi-objective hyperparameter optimization with performance uncertainty [62.997667081978825]
This paper presents results on multi-objective hyperparameter optimization with uncertainty on the evaluation of Machine Learning algorithms.
We combine the sampling strategy of Tree-structured Parzen Estimators (TPE) with the metamodel obtained after training a Gaussian Process Regression (GPR) with heterogeneous noise.
Experimental results on three analytical test functions and three ML problems show the improvement over multi-objective TPE and GPR.
arXiv Detail & Related papers (2022-09-09T14:58:43Z) - Multi-objective Pointer Network for Combinatorial Optimization [10.286195356515355]
Multi-objective optimization problems (MOCOPs) exist in various real applications.
Deep reinforcement learning (DRL) methods have been proposed to generate approximate optimal solutions to the optimization problems.
This study proposes a single-model deep reinforcement learning framework, called multi-objective Pointer Network (MOPN)
arXiv Detail & Related papers (2022-04-25T14:02:34Z) - A survey on multi-objective hyperparameter optimization algorithms for
Machine Learning [62.997667081978825]
This article presents a systematic survey of the literature published between 2014 and 2020 on multi-objective HPO algorithms.
We distinguish between metaheuristic-based algorithms, metamodel-based algorithms, and approaches using a mixture of both.
We also discuss the quality metrics used to compare multi-objective HPO procedures and present future research directions.
arXiv Detail & Related papers (2021-11-23T10:22:30Z) - Multi-objective Asynchronous Successive Halving [10.632606255280649]
We propose algorithms that extend successive asynchronous halving (ASHA) to the multi-objective (MO) setting.
Our empirical analysis shows that MO ASHA enables to perform MO HPO at scale.
Our algorithms establish new baselines for future research in the area.
arXiv Detail & Related papers (2021-06-23T19:39:31Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.