MO-DEHB: Evolutionary-based Hyperband for Multi-Objective Optimization

• URL: http://arxiv.org/abs/2305.04502v2
• Date: Thu, 11 May 2023 07:32:09 GMT
• Title: MO-DEHB: Evolutionary-based Hyperband for Multi-Objective Optimization
• Authors: Noor Awad, Ayushi Sharma, Philipp Muller, Janek Thomas and Frank Hutter
• Abstract summary: MO-DEHB is an effective and flexible multi-objective (MO) that extends the recent evolutionary Hyperband method DEHB. A comparative study against state-of-the-art MOs demonstrates that MO-DEHB clearly achieves the best performance across our 15 benchmarks.
• Score: 30.54386890506418
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Hyperparameter optimization (HPO) is a powerful technique for automating the tuning of machine learning (ML) models. However, in many real-world applications, accuracy is only one of multiple performance criteria that must be considered. Optimizing these objectives simultaneously on a complex and diverse search space remains a challenging task. In this paper, we propose MO-DEHB, an effective and flexible multi-objective (MO) optimizer that extends the recent evolutionary Hyperband method DEHB. We validate the performance of MO-DEHB using a comprehensive suite of 15 benchmarks consisting of diverse and challenging MO problems, including HPO, neural architecture search (NAS), and joint NAS and HPO, with objectives including accuracy, latency and algorithmic fairness. A comparative study against state-of-the-art MO optimizers demonstrates that MO-DEHB clearly achieves the best performance across our 15 benchmarks.

Related papers

• Progressive Multimodal Reasoning via Active Retrieval [64.74746997923967]
  Multi-step multimodal reasoning tasks pose significant challenges for large language models (MLLMs) We propose AR-MCTS, a universal framework designed to progressively improve the reasoning capabilities of MLLMs. We show that AR-MCTS can optimize sampling diversity and accuracy, yielding reliable multimodal reasoning.
  arXiv  Detail & Related papers  (2024-12-19T13:25:39Z)
• A Survey on Inference Optimization Techniques for Mixture of Experts Models [50.40325411764262]
  Large-scale Mixture of Experts (MoE) models offer enhanced model capacity and computational efficiency through conditional computation. deploying and running inference on these models presents significant challenges in computational resources, latency, and energy efficiency. This survey analyzes optimization techniques for MoE models across the entire system stack.
  arXiv  Detail & Related papers  (2024-12-18T14:11:15Z)
• MOANA: Multi-Objective Ant Nesting Algorithm for Optimization Problems [21.80971564725773]
  The Multi-Objective Ant Nesting Algorithm (MOANA) is a novel extension of the Ant Nesting Evolutionary Algorithm (ANA) MOANA addresses key limitations of traditional evolutionary algorithms by improving scalability and diversity in multi-objective scenarios. MOANA's applicability to real-world engineering optimization, such as welded beam design, showcases its ability to generate a broad range of optimal solutions.
  arXiv  Detail & Related papers  (2024-11-08T18:31:53Z)
• Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Merging [111.8456671452411]
  Multi-task learning (MTL) leverages a shared model to accomplish multiple tasks and facilitate knowledge transfer. We propose a Weight-Ensembling Mixture of Experts (WEMoE) method for multi-task model merging. We show that WEMoE and E-WEMoE outperform state-of-the-art (SOTA) model merging methods in terms of MTL performance, generalization, and robustness.
  arXiv  Detail & Related papers  (2024-10-29T07:16:31Z)
• Intuition-aware Mixture-of-Rank-1-Experts for Parameter Efficient Finetuning [50.73666458313015]
  Large Language Models (LLMs) have demonstrated significant potential in performing multiple tasks in multimedia applications. MoE has been emerged as a promising solution with its sparse architecture for effective task decoupling. Intuition-MoR1E achieves superior efficiency and 2.15% overall accuracy improvement across 14 public datasets.
  arXiv  Detail & Related papers  (2024-04-13T12:14:58Z)
• Multi-Objective Population Based Training [62.997667081978825]
  Population Based Training (PBT) is an efficient hyperparameter optimization algorithm. In this work, we introduce a multi-objective version of PBT, MO-PBT.
  arXiv  Detail & Related papers  (2023-06-02T10:54:24Z)
• BOtied: Multi-objective Bayesian optimization with tied multivariate ranks [33.414682601242006]
  In this paper, we show a natural connection between non-dominated solutions and the extreme quantile of the joint cumulative distribution function. Motivated by this link, we propose the Pareto-compliant CDF indicator and the associated acquisition function, BOtied. Our experiments on a variety of synthetic and real-world problems demonstrate that BOtied outperforms state-of-the-art MOBO acquisition functions.
  arXiv  Detail & Related papers  (2023-06-01T04:50:06Z)
• Speeding Up Multi-Objective Hyperparameter Optimization by Task Similarity-Based Meta-Learning for the Tree-Structured Parzen Estimator [37.553558410770314]
  In this paper, we extend TPE's acquisition function to the meta-learning setting using a task similarity defined by the overlap of top domains between tasks. In the experiments, we demonstrate that our method speeds up MO-TPE on tabular HPO benchmarks and attains state-of-the-art performance.
  arXiv  Detail & Related papers  (2022-12-13T17:33:02Z)
• Multi-objective Pointer Network for Combinatorial Optimization [10.286195356515355]
  Multi-objective optimization problems (MOCOPs) exist in various real applications. Deep reinforcement learning (DRL) methods have been proposed to generate approximate optimal solutions to the optimization problems. This study proposes a single-model deep reinforcement learning framework, called multi-objective Pointer Network (MOPN)
  arXiv  Detail & Related papers  (2022-04-25T14:02:34Z)
• A survey on multi-objective hyperparameter optimization algorithms for Machine Learning [62.997667081978825]
  This article presents a systematic survey of the literature published between 2014 and 2020 on multi-objective HPO algorithms. We distinguish between metaheuristic-based algorithms, metamodel-based algorithms, and approaches using a mixture of both. We also discuss the quality metrics used to compare multi-objective HPO procedures and present future research directions.
  arXiv  Detail & Related papers  (2021-11-23T10:22:30Z)
• Multi-objective Asynchronous Successive Halving [10.632606255280649]
  We propose algorithms that extend successive asynchronous halving (ASHA) to the multi-objective (MO) setting. Our empirical analysis shows that MO ASHA enables to perform MO HPO at scale. Our algorithms establish new baselines for future research in the area.
  arXiv  Detail & Related papers  (2021-06-23T19:39:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.